Skip to content

feat(agc): report device losses with VK_EXT_device_fault and dispatch checkpoints - #5453

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
overlordxrz-source:feat/agc-device-fault
Oct 14, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
overlordxrz-source:feat/agc-device-fault

Conversation

@overlordxrz-source

Copy link
Copy Markdown
Contributor

What

Port of @MichonGoddijn231849's 011e7c70 from his gta-v/testing branch, with him as the commit author. He okayed this port in #1203: "main has no device-loss reporting yet. On NVIDIA the fault info mostly gives instruction pointers it can't map to a shader, so the checkpoints are the useful part: which dispatch or draw was last started and which had finished."

  • Fault report. On a device with VK_EXT_device_fault and its deviceFault feature, the driver enables the extension. The first VK_ERROR_DEVICE_LOST that Graphics::Check or the device's own check sees then prints the fault description, each faulting address with its kind and precision, and the vendor fault codes as [device-fault] lines. The vendor binary is not requested.
  • Checkpoints. APS5_GPU_CHECKPOINTS=1, on a device with VK_NV_device_diagnostic_checkpoints, records a checkpoint with the program address before each compute dispatch. The report then lists the last checkpoints the GPU reached, which names the dispatch that hung.
  • Without either extension, no extension is enabled and nothing is recorded or printed. Checkpoints are off by default.

Port notes against Michon's commit (the hunks no longer applied after #639):

  • The checkpoint function lives in VulkanDevice::State, since recordDispatch is a member now. The device-lost hook is a std::function that captures the device, the queue and the two entry points. ~State clears it.
  • The device's own check (present path) reports a loss too.
  • The report is in Execution/include/DeviceFault.hpp so it can be tested. It bounds unterminated driver descriptions and the counts the second query returns.
  • APS5_GPU_CHECKPOINTS is documented in docs/dev/BUILD.md.

Why: Michon used this to diagnose the GPU hang in GTA V's ray tracing modes (#1203, open issue A). A device loss on main today only throws Vulkan result -4, with nothing about where it happened.

New test agc_device_fault (tests/DeviceFault.cpp) drives the report through mock vkGetDeviceFaultInfoEXT and vkGetQueueCheckpointDataNV. It covers:

  • the count query and then the data query, with no vendor binary requested
  • the description, address-kind and vendor lines, including an unterminated vendor description
  • a failed count query
  • checkpoint lines, and an empty checkpoint list
  • no output and no query when neither entry point is present

Tested

  • Linux aarch64 (zig 0.17.0 clang): agc_device_fault passes.
  • VulkanDevice.cpp, Resources.cpp (through Context.hpp) and the test compile for x86_64-linux-gnu and x86_64-windows-gnu. The test also links for x86_64-linux-gnu (not run, the host is aarch64).
  • Not tested on a real device loss, which needs a GPU hang.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none (diagnostics for the ray tracing GPU hang in Grand Theft Auto V, Grand Theft Auto V (PPSA04264) #1203)
  • GPUs tested: none here (no GPU in this environment); the report is covered by the mock-driven unit test only
  • AI-assisted: yes

@Sam-mon7

Copy link
Copy Markdown
Contributor

Tested on Windows x86_64, which this PR hasn't run yet: Windows 11 Pro 26200, MinGW-w64 GCC 15.2.0 (WinLibs UCRT r7), Release, NVIDIA RTX 4070 (driver 596.21), head ab478107f.

The test doesn't link on Windows: undefined reference to '__gxx_personality_v0'. Like the other agc test executables, it needs the project's libc and libkernel. On Windows it also needs libc's directory on PATH: with the link fixed but no PATH, the executable exits 127. With both changes it builds and passes:

target_link_libraries(agc_device_fault_tests PRIVATE libc libkernel)
...
    add_test(NAME agc_device_fault COMMAND agc_device_fault_tests)
    if(WIN32)
        set_tests_properties(agc_device_fault PROPERTIES ENVIRONMENT_MODIFICATION
            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
    endif()

With that change, agc_device_fault, agc_driver and agc_driver_graphics pass on the RTX 4070.

What the fault info gives on real device losses. Demon's Souls (PPSA01341), RTX 4070, on a local branch (not main) with a similar VK_EXT_device_fault report: the same two-call query, no vendor binary. It covered three natural device losses at character creation. The report on this driver:

[gpu-fault] device lost at vkQueueSubmit recorder; ; addresses 1/1, vendor entries 0/0
[gpu-fault] address type=2 reported=0x7f3660000 precision=0x1000

The description was empty and no vendor entries came back. The address was a data write (WRITE_INVALID) with 4 KiB precision, not an instruction pointer. Matched against a binding history, it was the last 64 KiB of a buffer that had been freed about two minutes earlier, which pointed at a GPU copy writing through a stale address. Aftermath decoded the same loss as a Copy Engine DMA page fault at that address. So on Ada, at least for this kind of fault, the fault info alone was enough to localize it, alongside what the checkpoints give.

One design point: DeviceLostHook() is one hook for the whole process. This title creates two devices, a windowless one first and then the windowed one that renders. The second overwrites the first's hook, so a loss on the first device would query the other device's fault info. ~State of either device also clears the hook for both. Here the rendering device is created last, so its losses would be reported, but if the windowless device were destroyed first, the hook would be gone. Clearing the hook only when it still belongs to this device, or keeping one hook per VkDevice, would avoid both problems.

I closed my #5460 (the same report, fewer call sites) in favour of this one.

AI-assisted: yes

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Two things before this can go in:

  • agc_device_fault_tests doesn't link on Windows; it needs libc and libkernel and libc's directory on PATH, as the other agc tests have (Sam-mon7's CMake above).
  • DeviceLostHook() is one hook for the process: a second device overwrites the first's, and ~State of either device clears it for both. Please keep the hook per VkDevice, or clear it only when it still belongs to the device being destroyed.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@adriadam10 adriadam10 self-assigned this Oct 14, 2026
… checkpoints

A device that supports VK_EXT_device_fault reports, on the first
VK_ERROR_DEVICE_LOST a check sees, the fault description, the faulting
addresses (invalid reads and writes, or the instruction pointers of waves
still running when the device hung) and the vendor fault codes.
APS5_GPU_CHECKPOINTS=1 records a VK_NV_device_diagnostic_checkpoints
marker with the program address before each guest dispatch and prints the
last markers the GPU reached, which names the dispatch that hung. Without
either extension nothing changes.

Test: agc_device_fault drives the report through mock
vkGetDeviceFaultInfoEXT and vkGetQueueCheckpointDataNV entry points.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: overlordxrz-source <overlordxrz-source@users.noreply.github.com>
@adriadam10

Copy link
Copy Markdown
Collaborator

Rebased onto current main (the conflict was only adjacency in VulkanDevice.cpp) and added the two points raised above, in the same commit:

  • agc_device_fault_tests now links libc libkernel, and the agc_device_fault test prepends libc's directory to PATH on Windows, as in @Sam-mon7's snippet.
  • The device-lost hook is only cleared in ~State when Graphics::DeviceLostOwner() is still that device, so destroying a windowless device no longer removes the rendering device's hook.

Built and ran the scoped tests (libSceAgcDriver, libkernel, relinker): agc_device_fault passes; agc_driver_tessellation (RADV segfault) and guest_current_cpu (load flake) fail the same way without this PR. Merging once the Windows build is green.

@adriadam10
adriadam10 force-pushed the feat/agc-device-fault branch from ab47810 to 1402c6c Compare October 14, 2026 12:02
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@adriadam10
adriadam10 merged commit beb7e94 into boykopovar:main Oct 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants