Skip to content

fix(agc): keep resident heap mirrors without host copies - #15296

Open
Prophet069 wants to merge 6 commits into
boykopovar:mainfrom
Prophet069:perf/agc-heap-mirror-host-copy-free
Open

Prophet069 wants to merge 6 commits into
boykopovar:mainfrom
Prophet069:perf/agc-heap-mirror-host-copy-free

Conversation

@Prophet069

Copy link
Copy Markdown
Contributor

What

A resident heap mirror kept a 16 MiB host copy of every slab, so building a multi-GiB mirror needed as much host memory again, and a rebuild while the old mirror was alive needed it twice. The host copies are gone.

A refresh now only marks the changed ranges (ResidentBuffer::MarkChanged, an interval map in which a later mark wins). The flush reads the marked ranges straight from guest memory into bounded 64 MiB staging segments, waits after every 256 MiB when the caller allows it, and fills newly bound or zeroed blocks on the device with vkCmdFillBuffer.

The write-back reference of each use is a device copy of the written ranges recorded after the uploads and before the shader (ResidentBuffer::RecordReference); the write-back compares the readback after the shader with it, so only bytes the shader changed reach the guest and CPU writes made meanwhile are kept. Non-resident heap mirrors keep their pooled host references.

Reference, upload and readback buffers count as kept bytes of the recorder, so its existing bound submits before they pile up, and references made inside a batch completion are recorded in order with the uploads.

Tests: partly written 64 KiB blocks, CPU writes made while a dispatch is in flight, and two dispatches writing the same range.

Tested

NVIDIA GeForce RTX 3080 Laptop GPU, driver 616.64, Windows 10 Pro 22H2 (19045.6466).
Focused tests on main (9090972) and on this branch with VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1, ctest -V -R "heap_mirror|^agc_driver_graphics$|^agc_driver_key_miss$|^agc_driver_bda_device$|^agc_driver_recorder$|native_multisample_depth|depth_storage|^agc_host_import_retire$|^agc_host_import_sparse$":

  • main: 17 of 20 selected tests passed (agc_host_import_sparse skips on this machine; the three agc_driver_depth_storage_* tests were not built in this run); validation layer loaded in both logs; VUID- errors: none.
  • this branch: 17 of 20 selected tests passed (agc_host_import_sparse skips on this machine; the three agc_driver_depth_storage_* tests were not built in this run), the same as main; VUID- errors: none.
  • agc_driver_resident_heap_mirror includes the new partial-block and in-flight CPU write cases.
  • Death Stranding 2 (PPSA02015), local integration branch with the GPU heap path pull requests: two of three runs hit the host memory limit while building ~6 GiB heap mirrors at world load (16 GB RAM machine); with this change the world loaded and the run reached its 45 minute limit in the opening cutscene.

Checklist

🤖 Generated with Claude Code

A heap mirror slab refused by a full device heap stopped the title. The slab
now falls back to host memory, as buffers and images already do, and a
refusal lowers the heap mirror budget, evicts down to it and retries; a
failed bind gives back the blocks it took.

A slab is refused before the device heap is exhausted: when all live slabs
would pass APS5_HEAP_MIRROR_MIB or when the device heap would keep less than
256 MiB of its reported budget, so the shrink and sweep path runs before
Windows pages device memory. A budget lowered by refusals is a soft target
a mirror may pass up to the configured budget when nothing can be evicted,
and the headroom refusal is not repeated within two seconds on one thread.

Blocks are taken from the fullest slab, so partly used slabs drain and are
freed. Nonzero scans remember each 64 KiB block with its write generation
and read only blocks written since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Nov 2, 2026

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — comments

The PR merges and builds, the heap-mirror tests pass, and the validation comparison shows no new VUIDs. But the new tests also pass with main's production code, and the new slab-refusal, budget-shrink and nonzero-scan-cache paths have no test.

What I ran

Head a64a9a4a280cb5be0f80bb01ce98812e56bcda6d, main 5cb015caaaaf83f8ee431414ee6a457b5b307d49, merged tree a5a36646c486d472781b0f1732a84113e9acecda (--no-ff), Debian 13 VM, GCC 14.2.0, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 5cb015caaaaf83f8ee431414ee6a457b5b307d49...a64a9a4a280cb5be0f80bb01ce98812e56bcda6d
python3 tools/check_conventions.py --base 5cb015caaaaf83f8ee431414ee6a457b5b307d49
ctest --test-dir build -j8 --test-load 32 --output-on-failure -R "heap_mirror|lod_clamp"
# negative control: the five production files from main
git show 5cb015caaa…:core/libs/prx/libSceAgcDriver/Graphics/{include/Context.hpp,include/GuestBufferMemory.hpp,include/ResidentMirror.hpp,src/GuestBufferMemory.cpp,src/ResidentMirror.cpp} > …
ninja -C build -j32 -l32 agc_driver_resident_heap_mirror_tests agc_driver_image_sample_lod_clamp_tests
ctest --test-dir build --output-on-failure -R "^agc_driver_resident_heap_mirror$|lod_clamp_shader_lane"
# restore the merged files from a5a36646c4…, rebuild everything
ninja -C build -j32 -l32
git status --porcelain

Results

Check Result
git diff --check clean
check_conventions.py All conventions checks passed
Configure / build / libs (script) ok
agc_driver_resident_heap_mirror (with the new PartialBlockTests and ConcurrentWriteTests) passed
agc_heap_mirror_*, agc_sparse_heap_mirror_* (10 tests) passed
agc_driver_image_sample_lod_clamp_shader_lane (listed by the script as a new failure) fails, and fails the same way with main's production files, so it comes from main (baseline was the older 096c45570f), not from this PR
unwrap_self_..._destination_directory_group (listed as a new failure) a Python tool test failing on os.chown in a temporary directory (EINVAL); this PR changes no tool or relinker file, so it is the environment, not the PR
Validation layer (script, 759 tests) layer loaded in 758, missing in none, 0 tests with new VUID- lines
Worktree after restore git status --porcelain empty
lod_clamp_shader_lane output (same on merged tree and with main's production files)
AGC graphics: image_sample_lz through the S# that holds the LOD at 0 with T# MIN_LOD 768 over BASE_LEVEL 0 with a second S# of words 1 0 and 2 5242880, and a second T# of MIN_LOD 0 and LAST_LEVEL 0: thread 0 component 1 is 0.000000, expected 0.062500

Negative control

I put back main's versions of Context.hpp, GuestBufferMemory.hpp, ResidentMirror.hpp, GuestBufferMemory.cpp and ResidentMirror.cpp, kept the PR's tests/execution/ResidentHeapMirror.cpp, and rebuilt. agc_driver_resident_heap_mirror still passed (2.23 s).

That can be expected: the new partial-block, in-flight CPU write and double-dispatch cases describe behaviour that main's host-copy design already had. So they guard against regressions in the rewrite, but they don't show the change itself. Nothing tests the host-memory saving, the slab refusal, the budget shrink-and-retry loop or the nonzero-scan cache. I then put the merged files back, rebuilt the whole tree, and checked that the worktree is clean.

Findings

Blocking

None.

Non-blocking

  1. No test reaches the slab-refusal and budget-shrink path. ResidentMemory::refuseSlab (core/libs/prx/libSceAgcDriver/Graphics/src/ResidentMirror.cpp:176), shrinkHeapMirrorBudget (core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp:2814) and the 8-attempt retry loops (GuestBufferMemory.cpp:2849, GuestBufferMemory.cpp:3387) are the fix the PR depends on for PPSA02015. They run only when the device heap fills or when slabs pass APS5_HEAP_MIRROR_MIB. The existing tests pass on main without any of this code. Suggested fix: add a case that sets a small budget with SetHeapMirrorBudget (which HeapMirrorSlabBudget already reads). It should check that a slab past it is refused, the budget is lowered, an older mirror is evicted, and the new mirror is still built.
  2. The lowered budget never goes back up. heapMirrorMemoryLimit (GuestBufferMemory.cpp:2714) is only ever lowered, at GuestBufferMemory.cpp:2824, and nothing raises it again. So one refusal caused by a temporary peak in device-heap use leaves heapMirrorBudget() lowered for the rest of the process. The soft-target escape in reserveHeapBytes/growResident only applies on a retry, after a sweep has already evicted mirrors. So after one temporary peak, mirrors keep being evicted early for the whole session. Suggested fix: raise the limit again once the reported heap has headroom, or say in the description that the drop is permanent by design.
  3. The refusal is recognised by matching the error text. shrinkHeapMirrorBudget tests error.what() for "of a heap mirror slab". A change to either message (refuseSlab or allocate's operation string) would silently turn off the retry and bring back the stop this PR fixes. Suggested fix: use a dedicated exception type derived from std::runtime_error, or a flag returned by Take.
  4. Context::HeapReport::Fits is never used (core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp:340). refuseSlab does its own headroom arithmetic. Either use Fits or remove it.
  5. The nonzero-scan cache is shared global state. GuestNonzeroBlocks now keeps a process-wide unordered_map of up to 4M entries (ResidentMirror.cpp:379-380) behind one mutex, which it takes once or twice per 64 KiB block. The whole map is dropped when it fills (ResidentMirror.cpp:408). Its correctness depends on UnchangedSince coming back false for pages that were unmapped and mapped again (ForgetPages). I didn't find a test of that for this cache (ReusedAddressTests passes, but it also passed before the cache existed). This is the third separate change in the first commit; see the next item.
  6. One topic per PR. Besides dropping the host copies, the PR also includes a cherry-picked commit by another author (8a30d9a70f8b, "store a resident heap mirror's write-back from its readback buffer"), the slab-refusal and fallback work, fullest-slab allocation, and the scan cache (1743177ae4af). The checklist names Depends on: #15295, #12827. If those PRs carry some of these commits, the commits would be merged twice or conflict. Please state which commits belong to which PR, or move the separate fixes into their own PRs.

Not verified

  • The claimed host-memory saving and the PPSA02015 world-load result: I launched no title. lavapipe has a 64 GiB host-backed heap, so the device-heap headroom refusal and the host-memory fallback never run here.
  • The Windows/NVIDIA validation run in the description: I only ran on lavapipe.
  • The contents of #15295 and #12827: no context was available, so I couldn't check the overlap described in finding 6.
  • Whether a RecordStagingCopies flush that uploads newly marked ranges after takeHeapReferences recorded the device reference could put those bytes into the write-back. That would happen if the range differs between the reference and the readback without any shader store. I didn't build a test for this ordering. The new ConcurrentWriteTests doesn't cover it.

To get this to a good state

  • Add a test that reaches the refusal, shrink, sweep and retry path (finding 1). Ideally also add one for the scan cache across an unmap and remap at the same address (finding 5).
  • Decide whether the lowered budget should recover (finding 2), and replace the error-text match (finding 3).
  • Remove or use HeapReport::Fits (finding 4).
  • Clarify how this PR's commits relate to #15295 and #12827, or split the separate fixes out (finding 6).

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head a64a9a4 merged onto main 9090972 · tested 2026-11-03 00:39 UTC · 32 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,642 passed · 5 failed (0 new, 2 also fail on main, 3 unstable on main) · 44 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,691 of 3,607 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (974 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s, ⚠️ agc_driver_graphics_maps_fallback 30.9 s
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 3 of the last 10 main runs
agc_submit_progress: failed in 7 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 6 of the last 10 main runs
Failing on main too (2, not counted against this PR)
agc_driver_image_table
guest_socket_options_multicast
Validation IDs already present on main (4 IDs in 517 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (517 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 509 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
Skipped (44)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fps_limit: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: 
Reproduce
git fetch origin a64a9a4a280cb5be0f80bb01ce98812e56bcda6d 9090972b7e91fd266819cbd77a8b058e177b5682
git checkout 9090972b7e91fd266819cbd77a8b058e177b5682 && git merge --no-ff a64a9a4a280cb5be0f80bb01ce98812e56bcda6d
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Hi Prophet069,

This head contains the exact commit from #15295 plus four follow-up commits, while its checklist says Depends on: none. Could you declare #15295 and the intended landing order, or split that prerequisite out of this topic?

Please also settle the overlap/order with open #12827 (the same readback-vector-copy removal) and #12840 (BDA heap-store work). At their current heads, combining this PR with #12827 conflicts in GuestBufferMemory.cpp; #12840 conflicts in that file and GuestBufferMemory.hpp. A clear supersession/order and a rebase of the affected successor would keep the broader host-copy change and the BDA work from overwriting each other's reference/write-back handling.

@github-actions github-actions Bot added the agc libSceAgc* label Nov 3, 2026
@oneandonlydean oneandonlydean self-assigned this Nov 3, 2026
Prophet069 and others added 5 commits November 3, 2026 05:48
…ost fallback

A slab refused for headroom or for APS5_HEAP_MIRROR_MIB while the heap mirrors
held 256 MiB or less stopped the title: the budget shrink has nothing to evict
there. Refuse a slab only while more than that is held, so the allocation and
its host-memory fallback run otherwise. Refusals and failed slab allocations
throw HeapMirrorSlabRefused instead of being matched by their text, a budget
lowered by a refusal recovers after 30 s, and the headroom check uses
HeapReport::Fits. A test reports the device heap full and checks that small
mirrors are still made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k buffer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c12a20f)
A resident heap mirror kept a 16 MiB host copy per slab, so rebuilding a
multi-GB mirror needed as much host memory again. Refreshes now only mark
the changed ranges, the flush reads them straight from guest memory into
bounded staging and zero-fills new blocks on the device, and the write-back
reference of each use is a device copy recorded after the uploads and
before the shader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… references inside completions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…waves

RecordStagingCopies flushed changed resident heap mirror blocks without
allowing a wait, so a multi-GiB mirror upload held every 64 MiB staging
segment until the batch finished, gigabytes of host memory at once. It
now waits after each 256 MiB wave like UploadFinish does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prophet069
Prophet069 force-pushed the perf/agc-heap-mirror-host-copy-free branch from a64a9a4 to 7bbf091 Compare November 3, 2026 05:07
@Prophet069

Copy link
Copy Markdown
Contributor Author

Thanks. This PR is stacked: it builds on #15295 (its first two commits) and on #12827 (the cherry-picked write-back commit); its own commits are "drop the host copies of resident heap mirrors", "count heap mirror transfer buffers as kept bytes and record references inside completions" and "let staging-copy recording wait between heap mirror upload waves". The checklist says Depends on: #15295, #12827; it lands after both, and I rebase it when they move. It is rebased onto the reworked #15295 now.

On the review points: the refusal type, the budget recovery and HeapReport::Fits are addressed in #15295 (text match replaced by HeapMirrorSlabRefused, 30 s recovery, Fits used), with a test that reaches the refusal and fails without the fix. I'll look at the RecordStagingCopies ordering question (a flush after the reference was recorded) next and add a test for it.

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Hi Prophet069,

Thanks for declaring Depends on: #15295, #12827 and for the rewrite at 7bbf091cf8. Two points remain.

  1. fix(agc): write back BDA stores into writable heap mirrors #12840 (BDA stores into writable heap mirrors) is not mentioned in the PR. Merged with this head, its head 9fd023a310 still conflicts in core/libs/prx/libSceAgcDriver/Graphics/include/GuestBufferMemory.hpp and core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp, and both PRs change how takeHeapReferences and WriteBack handle the reference of a written heap range. Could you say which of the two lands first and which one is rebased on the other?

  2. RecordStagingCopies still calls ResidentBuffer::FlushChanged(context, true) (GuestBufferMemory.cpp:6916), and takeHeapReferences has already recorded the device reference copies by then (GuestBufferMemory.cpp:6046 and :6703). FlushChanged uploads every marked range of every resident buffer, so a range marked between the two calls (a refresh of the same heap by another buffer memory) would be uploaded after the reference was copied. The write-back would then see those bytes as changed by the shader and store them over CPU writes made since. You mentioned looking at this ordering; could you show that no mark can land between the two calls, or move the flush ahead of the reference copies, and add a test for it?

@EagleBearerYS

Copy link
Copy Markdown
Contributor

@Prophet069, on the order with #12840 (BDA stores into writable heap mirrors), which oneandonlydean asked about above: #12840 can go after this PR.

#12840's checklist will say Depends on: #15296 (merge order only).

@Prophet069

Copy link
Copy Markdown
Contributor Author

Thanks, agreed: #15296 lands first (after #15295 and #12827, which it is stacked on), and #12840 rebases onto it and takes resident mirrors' references from RecordReference.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 7bbf091 merged onto main f6232fc · tested 2026-11-03 05:53 UTC · 34 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,678 passed · 6 failed (0 new, 3 also fail on main, 3 unstable on main) · 49 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,733 of 3,743 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (934 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.2 s
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 4 of the last 10 main runs
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 5 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (4 IDs in 525 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (525 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 517 more
VUID-vkCmdCopyImage-srcImage-01548 (8 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth, agc_driver_stencil_null_feedback
VUID-vkCmdCopyImage-srcImage-01551 (8 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth, agc_driver_stencil_null_feedback
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
Skipped (49)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver
Reproduce
git fetch origin 7bbf091cf83a767b16e38f966d2d72d9321b0d81 f6232fcfda70644495273f6b8dd88cf0603463d6
git checkout f6232fcfda70644495273f6b8dd88cf0603463d6 && git merge --no-ff 7bbf091cf83a767b16e38f966d2d72d9321b0d81
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 7bbf091 merged onto main 7feb37e · tested 2026-11-03 06:00 UTC · 37 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,670 passed · 10 failed (0 new, 9 also fail on main, 1 unstable on main) · 54 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,734 of 3,743 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (992 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.2 s, ⚠️ public_headers 84.3 s, ⚠️ check_skips 39.9 s, ⚠️ agc_driver_image_sample_unnormalized 37.9 s, ⚠️ agc_driver_lazy_resolved_abi 34.1 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 5 of the last 7 main runs
Failing on main too (9, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_depth_buffer_clear
agc_driver_native_depth_clears
agc_idle_publish_wait
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (56, not in the main baseline's validation pass)
public_headers_compile_0
public_headers_compile_10
public_headers_compile_12
public_headers_compile_14
public_headers_compile_16
public_headers_compile_18
public_headers_compile_19
public_headers_compile_2
public_headers_compile_20
public_headers_compile_21
public_headers_compile_22
public_headers_compile_23
public_headers_compile_24
public_headers_compile_25
public_headers_compile_26
public_headers_compile_27
public_headers_compile_28
public_headers_compile_29
public_headers_compile_30
public_headers_compile_31
public_headers_compile_32
public_headers_compile_33
public_headers_compile_34
public_headers_compile_35
public_headers_compile_36
public_headers_compile_37
public_headers_compile_38
public_headers_compile_39
public_headers_compile_4
public_headers_compile_40
public_headers_compile_41
public_headers_compile_42
public_headers_compile_43
public_headers_compile_44
public_headers_compile_45
public_headers_compile_46
public_headers_compile_47
public_headers_compile_48
public_headers_compile_49
public_headers_compile_50
public_headers_compile_51
public_headers_compile_52
public_headers_compile_53
public_headers_compile_54
public_headers_compile_55
public_headers_compile_56
public_headers_compile_57
public_headers_compile_58
public_headers_compile_59
public_headers_compile_6
public_headers_compile_60
public_headers_compile_61
public_headers_compile_62
public_headers_compile_63
public_headers_compile_8
trace_digest_TraceDigestTests_test_cli_rejects_invalid_event_fields_without_partial_output
Validation IDs already present on main (4 IDs in 519 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (519 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 511 more
VUID-vkCmdCopyImage-srcImage-01548 (6 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (6 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
Skipped (54)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) o
Reproduce
git fetch origin 7bbf091cf83a767b16e38f966d2d72d9321b0d81 7feb37ee3e30e10139b483fb4df689d5b5b69ab8
git checkout 7feb37ee3e30e10139b483fb4df689d5b5b69ab8 && git merge --no-ff 7bbf091cf83a767b16e38f966d2d72d9321b0d81
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — approved

The new commit 89e3292072 ("let a small heap mirror reach the slab allocation and its host fallback") does four things. It replaces the error-text match with a HeapMirrorSlabRefused type. It lets a lowered budget recover after 30 s. It uses HeapReport::Fits. It refuses a slab only while the heap mirrors hold more than 256 MiB. Its new FullDeviceHeapTests case fails when that last guard is removed. The other commits were rebased onto it with the same content.

What changed since my last review

  • New commit 89e329207201, between the slab-refusal commit and the cherry-picked write-back commit. The later commits were rebased onto it. Per the range-diff, they only change around the new test struct and the try/catch in ResidentMemory::Take. The head commit 7bbf091cf8 is unchanged.
  • The author explained in discussion how the commits split across PRs. The first two commits are #15295, 7c184fb3ae is the cherry-pick from #12827, and the last three are this PR's own. The checklist says Depends on: #15295, #12827. The author also agreed with EagleBearerYS that #12840 lands after this PR.

What I ran

Head 7bbf091cf83a767b16e38f966d2d72d9321b0d81, main d1ae628bc2f62459cce1e9872ba5632c938c71f2, merged tree bf1ec647053b1866f995446eb40a85dcac8f0323 (--no-ff). Debian 13 VM, GCC 14.2.0, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check d1ae628bc2f62459cce1e9872ba5632c938c71f2...7bbf091cf83a767b16e38f966d2d72d9321b0d81
python3 tools/check_conventions.py --base d1ae628bc2f62459cce1e9872ba5632c938c71f2
ctest --test-dir build -j8 --test-load 32 --output-on-failure -R "heap_mirror"
# negative control: delete `if (live <= Reserve) return;` from ResidentMemory::refuseSlab
sed -i "/if (live <= Reserve) return;/d" core/libs/prx/libSceAgcDriver/Graphics/src/ResidentMirror.cpp
ninja -C build -j32 -l32 agc_driver_resident_heap_mirror_tests
ctest --test-dir build --output-on-failure -R "^agc_driver_resident_heap_mirror$"
# restore
git show bf1ec647053b1866f995446eb40a85dcac8f0323:core/libs/prx/libSceAgcDriver/Graphics/src/ResidentMirror.cpp > core/libs/prx/libSceAgcDriver/Graphics/src/ResidentMirror.cpp
ninja -C build -j32 -l32
ctest --test-dir build --output-on-failure -R "^agc_driver_resident_heap_mirror$"
git status --porcelain
Check Result
git diff --check clean
check_conventions.py All conventions checks passed
Configure / build / libs (script) ok
heap_mirror tests (10, including agc_driver_resident_heap_mirror with FullDeviceHeapTests) all passed (2.5 s)
Validation layer (script, 767 tests) layer loaded in 766, missing in none, 0 tests with new VUID- lines
unwrap_self_..._destination_directory_group (script: new failure) os.chown on a temporary directory fails with EINVAL. This PR changes no tool file, so the cause is the environment, as in my last review
Negative control (guard removed) agc_driver_resident_heap_mirror fails
After restore test passes, git status --porcelain empty
Negative control output
[heap-mirror] refused: slabs 0 (0 MiB), free slots 0 MiB, live mirrors 0 bound 0 MiB, budget 24576 MiB
AGC graphics: heap mirror of 0x7f7b55800000+0x2000000: vkAllocateMemory of a heap mirror slab refused: its device heap would keep less than 256 MiB of its reported budget; heap mirrors hold 15 MiB of 24576 MiB (APS5_HEAP_MIRROR_MIB)

I didn't do a full restore of main's production files. The new test calls SetDeviceHeapFullForTests, which doesn't exist on main, so it wouldn't compile. Removing only the guard is the narrower control, and it shows the new test exercises the headroom refusal.

Previous findings

  1. No test reaches the refusal/shrink path. Partly resolved, still open for the shrink. FullDeviceHeapTests (core/libs/prx/libSceAgcDriver/tests/execution/ResidentHeapMirror.cpp:370) now reaches refuseSlab (see the negative control). But it holds less than 256 MiB, so shrinkHeapMirrorBudget returns at core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp:2841. The lowering, the eviction sweep and the 8-attempt retry are still not run by any test. Non-blocking.
  2. The lowered budget never recovers. Resolved. heapMirrorBudget puts the limit back after 30 s (GuestBufferMemory.cpp:2734), and the time a refusal lowers it is stored at GuestBufferMemory.cpp:2842. No test covers it.
  3. Refusal matched by error text. Resolved. It is now dynamic_cast<const HeapMirrorSlabRefused*> (GuestBufferMemory.cpp:2834). Both the refusal and a failed slab allocation throw that type (core/libs/prx/libSceAgcDriver/Graphics/src/ResidentMirror.cpp:155-159, :199). That is the same set of errors the old text match caught.
  4. HeapReport::Fits unused. Resolved. It is used at ResidentMirror.cpp:193.
  5. Nonzero-scan cache is shared global state, with no test across unmap and remap. Still open, unchanged (ResidentMirror.cpp:381). Non-blocking.
  6. One topic per PR. Resolved by declaration. The commits are attributed to #15295 and #12827 in discussion, and the checklist says Depends on: #15295, #12827. The landing order is author-reported. I couldn't check the other PRs' heads.

New findings

Blocking

None.

Non-blocking

  1. A flush after the reference copy is still possible. oneandonlydean raised this, and it was under "Not verified" in my last review. The author said they would look at it next, and the head commit hasn't changed. I can confirm the order this would happen in. recordTransfer records uploads into recorder->Commands() in sequence (ResidentMirror.cpp:61-68), and so does RecordReference (ResidentMirror.cpp:616-627). So any range still marked when RecordStagingCopies calls ResidentBuffer::FlushChanged(context, true) (GuestBufferMemory.cpp:6928) is uploaded after the reference that takeHeapReferences copied (GuestBufferMemory.cpp:6715-6718). The write-back would then treat it as a shader write. I didn't build a case that puts a mark between the two calls. Suggested fix: show that no mark can land there, or make the write-back ignore ranges flushed after the reference. Add a test either way.
  2. Recovery is by time only. After 30 s the limit goes back to unlimited, whether or not the device heap has room again (GuestBufferMemory.cpp:2734). On a heap that stays full, this can mean another refuse, shrink and evict cycle every 30 s. That is acceptable as a design choice, but a short note in the commit or description would help.

Not verified

  • The host-memory saving and the PPSA02015 world-load result: I launched no title.
  • The NVIDIA/Windows run in the description (author-reported). lavapipe reports a large host-backed heap, so the real headroom refusal and the host fallback only ran through the test override.
  • The shrink, evict and retry loop, and the 30 s recovery: no test reaches either, and I didn't write one.
  • The contents and landing order of #15295, #12827 and #12840.

@sktharunbalaji13-cmd

Copy link
Copy Markdown
Contributor

A real-world data point for this PR, from Ghost of Tsushima DC (PPSA02225) on a 16 GB RAM / 8 GB VRAM Windows laptop (RTX 4060 Laptop), main b11c7bf + #12840 + #12792.

  • What the host copies cost there. A VirtualQueryEx size histogram of the process's host-side allocations at 200 s shows 149 x 16 MiB = 2384 MiB of private memory, the slab host copies (ResidentMirror.cpp:124-129). At the same time the census reports heap mirrors hold ~1 GB. The rest of the host side is 43 x 32 MiB staging chunks, 8 MiB unit-shadow slabs and buffer-pool blocks, about 7.2 GB in all.
  • Why it matters. The guest band is a flat 9.9 GB. With these host copies on top, RAM is full from ~15 s. Every address-based build then re-copies 63k-100k changed 64 KiB blocks of the two big heaps from the pagefile, and the frame loop freezes 8-17 s. That freeze is why the title's intro movie (slaved to its Wwise soundtrack) skips the PlayStation logo on this machine. With 8.9 GB free after a reboot, the logo animation starts to play.
  • Details and screenshots: Ghost of Tsushima Director's Cut (PPSA02225, Windows): black frames despite continuing draw and presentation activity #5166 (comment)

I'll build this PR on top of that tree and report the private-memory, freeze and logo results on the same machine.

@EagleBearerYS

EagleBearerYS commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

@sktharunbalaji13-cmd a note before you build it: this PR and #12840 don't merge together yet. Both change GuestBufferMemory.hpp and GuestBufferMemory.cpp, and #12840's resident write-back reads the resident host copies (ResidentBuffer::Block and Read) that this PR removes. The planned order is this PR first, then #12840 rebased off the host copies (note).

@sktharunbalaji13-cmd

Copy link
Copy Markdown
Contributor

@EagleBearerYS thanks, that matches what I saw: this PR doesn't apply on top of #12840 (the write-back and resident-copy hunks are rejected). So I'm testing the two memory PRs separately on the same 16 GB / 8 GB machine:

Each is compared with that tree's run, on the same machine and in the same conditions: free RAM and private bytes over time, freeze length after the intro soundtrack starts, how far the intro gets, and whether the 135 s VRAM abort still happens. I'll post the numbers here and on #15295.

@sktharunbalaji13-cmd

Copy link
Copy Markdown
Contributor

Results from Ghost of Tsushima DC (PPSA02225) on the 16 GB RAM / 8 GB VRAM laptop (RTX 4060 Laptop, driver 617.42).

This PR was applied onto main b11c7bf, without #12840 as you advised. The only manual resolution: the WriteBack hunk keeps main's newer storeChanged(..., GuestMemory::WriteChanged) argument, and the duplicate HeapReport hunk was dropped because main already has it. Every run uses the same relinked executable and environment (APS5_SKIP_REFUSED_DRAWS=1 plus an AudioOut2 trace), 150 s, with a screenshot every 2 s. The conditions were the same for all of them: Claude open, Defender real-time protection on.

main + #12840 + #12792, run 1 same, run 2 + #15295 main + #15296 (no #12840)
free RAM at launch 8.1 GB 10.0 GB 10.9 GB 10.8 GB
first freeze after the intro soundtrack starts 18.1 s 14.0 s 11.9 s 12.0 s
peak process private memory 25.6 GB 24.6 GB 25.1 GB 21.8 GB
lowest free RAM 3 MB 54 MB 9 MB 99 MB
VRAM wall no abort in 150 s abort at ~120 s: vkAllocateMemory of a heap mirror slab ... -2; heap 0 7852/7188 MiB no abort; device memory refused a heap mirror slab; heap mirror budget lowered x9 no abort; the same message x10
intro reached card 36-38 s card 28-34 s card 26-34 s card 26-34 s
  • Memory: this PR lowers the peak private memory by 2.8-3.8 GB, in line with the 2.4 GB of slab host copies we measured, plus bounded staging.
  • VRAM: with this branch the device heap filling up lowers the mirror budget instead of aborting. The base tree aborted there in 2 of its last 3 runs (at ~120 s here, and at 135 s in the run linked on Ghost of Tsushima Director's Cut (PPSA02225, Windows): black frames despite continuing draw and presentation activity #5166).
  • Freeze: shorter than the base's 14-18 s, but still ~12 s, because RAM still fills up (lowest free 99 MB). So the logo still depends on how much RAM the machine has free. In these runs (Claude and Defender running) the intro reached the full-brightness card but no logo frame was captured. The earlier clean run with everything closed and Defender off caught the logo's first frame at 34 s.

Logs are available on request.

@sktharunbalaji13-cmd

Copy link
Copy Markdown
Contributor

One more data point on the same 16 GB RAM / 8 GB VRAM laptop: main b11c7bf + this PR, PPSA02225, nothing else open, Defender off, 11.3 GB free at launch.

The VRAM abort from the base tree is gone, but after the budget was lowered this run stopped at 147 s with a different abort:

[gpu] device memory refused a heap mirror slab; heap mirror budget lowered to 1760 MiB
[gpu] device memory refused a heap mirror slab; heap mirror budget lowered to 1274 MiB
[gpu] device memory refused a heap mirror slab; heap mirror budget lowered to 1296 MiB
[heap-mirror] shrank: slabs 32 (512 MiB), free slots 37 MiB, live mirrors 1 bound 298 MiB, budget 24576 MiB
FATAL: an address-based build copies 4696 MiB of 1 registered ranges, past 64 MiB (APS5_ADDRESS_COPY_MAX_MIB); the largest: 0x1200000000+0x125800000 (4696.0 MiB committed, import over the host import budget, heap mirror refused)

The 4.7 GB heap 0x1200000000 is over the 3.9 GB host-import budget, and once the lowered budget refuses its mirror, the only path left is the copy, which APS5_ADDRESS_COPY_MAX_MIB (64 MiB) forbids, so the build aborts. On this machine the lowered budget can therefore turn a recoverable shortage into a stop.

Is that path meant to evict and retry, or to let the mirror pass the lowered budget? Your description says a lowered budget is a soft target that a mirror may pass when nothing can be evicted.

The earlier run of this PR on the same machine (no abort in 150 s) is in the comment above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* needs rebase Conflicts with main, rebase needed unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants