Repository navigation
perf(agc): draws that only store to a buffer continue one render pass - #7340
Conversation
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
Thanks, the tracker change itself looks right. Write-only still conflicts with reads, read-write uses and unknown accesses in both directions, and the classification leaves out atomic and read bindings and ranges shared with other allocations. One ordering case needs handling before this can go in. On this driver, guest sync packets never reach the recorder. Could you seal the pass's write-only set when queue 0 sees a partial or cache flush EVENT_WRITE or an ACQUIRE_MEM while a pass is open? Any later write-only store to those ranges would then be a WriteAfterWrite. For example, a If you have it, a pass-count or write-only-draw counter (or the frame numbers from the UE4 title) would also show how often this triggers. |
github-actions
Bot
added
perf
A guest buffer the recompiler classifies as only stored to (bufferWritten, not bufferRead, not bufferAtomic), which no other allocation of the draw overlaps, is a write-only range of the draw. It conflicts with every read and every other write of the open render pass, but not with the write-only stores of other draws to the same range: the guest GPU leaves stores of different draws unordered as well, since no barrier separates them. Before, such a buffer counted as both read and written, so consecutive draws storing to it each began a new pass with two full barriers. APS5_NO_WRITE_ONLY_PASSES=1 restores that. Unreal Engine 4 stores virtual texture feedback from every base pass material that samples a virtual texture into one buffer per frame: in Stray (PPSA02100) about 90 % of the pass restarts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Queue 0 skips partial and cache flush EVENT_WRITE packets and ACQUIRE_MEM, and nothing ends the open render pass on them, so the pass hazard tracker is all that orders two draws inside one pass. A write-only store, a flush and a second write-only store to the same bytes stayed in one pass with no barrier between them, although the flush orders them on the guest GPU. At such a packet, the write-only ranges of the open pass now count as written, so a later write-only store to them is a WriteAfterWrite and begins a new pass. The seal is queued behind the draws in the draw pipeline (or made at once when it is idle), once per run of flushes after a draw. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
95e7314 to
12d720f
Compare
|
This pull request is labeled |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@oneandonlydean thanks, done in 12d720f. The branch is rebased onto
How often it triggers, Stray (PPSA02100) on Frame p50, alternating runs on the same save ( AI-assisted: yes (implementation, analysis and text with Claude Code; the runs are from my machine). |
github-actions
Bot
added
docs
needs rebase
|
Rebase handoff — rebased onto current Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, To pick it up: Head is |
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp # core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
|
@blawesom thanks for the rebase and the full suite run. @oneandonlydean merged AI-assisted: yes (text with Claude Code). |
github-actions
Bot
removed
the
unblocks title
|
A data point at the stock settings, since this has been open for a few days. On my local stack ( The branch still merges with AI-assisted: yes (text with Claude Code; the runs are from my machine). |
3 participants
What
A guest buffer the recompiler classifies as only stored to (
bufferWritten, notbufferRead, notbufferAtomic), which no other allocation of the draw overlaps, is now a write-only range of the draw (PassAccess::writeOnly). It conflicts with every read and every other write of the open render pass, but not with the write-only stores of other draws to the same range. The guest GPU leaves stores of different draws to such a buffer unordered as well, since no barrier separates them.Before, such a buffer counted as both read and written, so consecutive draws storing to it each began a new render pass with two full barriers. Unreal Engine 4 stores virtual texture feedback from every base-pass material that samples a virtual texture into one buffer per frame. In Stray (PPSA02100) this change takes the render passes begun from 33.8 % to 7.5 % of draws.
PassHazards: a write-only range conflicts with reads (read after write), with read-write uses (write after write) and with unknown reads or writes, in both directions; write-only ranges of different draws over the same bytes do not conflict.PassHazards::SealWriteOnly()(throughVulkanDevice::SealPassWriteOnlyandRecorder::SealPassWriteOnly) moves the open pass's write-only ranges into its writes, so a later write-only store to the same bytes begins a new pass. While the draw pipeline has draws in flight, the seal is queued behind them.ShaderResourcesmarks a written guest buffer allocation write-only when its binding element is stored to, neither read nor atomic, and no other allocation of the build overlaps it (WriteOnlyRanges).Draw.cpptakes those ranges out of the draw's own buffer reads and writes that it hands toPassHazards, and passes them as the draw's write-only ranges, unless the draw has unknown memory accesses. Indirect argument ranges and any other read added to the pass reads stay reads.APS5_NO_WRITE_ONLY_PASSES=1restores the previous behaviour (listed indocs/dev/ENVIRONMENT.md).agc_pass_hazards_testscovers write-only stores continuing the pass, and a read, a read-write use, unknown reads and unknown writes after them, in either order. For the seal, it checks that a write-only store over sealed write-only stores is a write after write, that write-only stores elsewhere still continue the pass, and thatClearremoves the seal.Based on
main0bc3003, which has #7184 (a draw that only reads through the BDA table continues its pass); the two changes share thePassAccessline ofrecordDraw. The rebase from f1af6c5 had one conflict, next to the packet counter that #6012 added afterlandPendingWaitsinPacketExecution.cpp; both lines stay. A draw with unknown memory accesses, BDA reads included, keeps its stored buffers as read and written, as before, and unknown reads conflict with write-only ranges in either order. No open PR changesPassHazards.tools/pr_overlap.pyat this head finds changed lines shared with #5431 (two lines ofDraw.cppnext to the pass hazard check), #6566 (thepassReadsline ofDraw.cpp) and #7618 (two lines ofShaderResources.cpp). #7618 stages write-only buffer elements of sweeping dispatches; it doesn't touch render passes orPassHazards. It couldn't compare 60 open PRs that share the changed files, because they conflict withmain; none of them implements write-only passes or a flush seal. #6175 adds in-place draw inputs to the pass reads; they stay reads with this change.Tested
Linux (CachyOS, kernel 7.2.10), Ryzen 9 9950X, RTX 3090 (NVIDIA 615.78.08), GCC 16.2.1, Release, 3840x2160.
agc_pass_hazards_testspasses at 12d720f (onmain0bc3003; the head, d840b10, only adds thedocs/dev/ENVIRONMENT.mdrow), and so do 721 of the 723agc_tests (ctest -R '^agc_';agc_driver_stencil_ref_exportis skipped;agc_driver_bda_deviceloses the device with an NVIDIA Xid 109 here, and does the same onmain0bc3003 without this change, while it passed on f1af6c5 (agc_driver_bda_device times out in vkWaitForFences since the BDA anchor lookup (#5859, d422c760c) #7584, from perf(shader): look up the BDA range of a user-data pointer once at program entry #5859's d422c76; fix(shader): leave the BDA lookup functions through one return #7571 fixes it)).tools/check_conventions.py --base origin/mainpasses.mainf1af6c5.engine.ini, one run at a time withmainand this change alternating, after one warm-up run of each build. Trace summaries withAPS5_TRACE_SECONDS=185:20and no otherAPS5_TRACE_*/APS5_PROFILE_*variables; the runner redirects the log to a file. Queue worker 0's event buffer fills after 4 to 5 s in every run (trace_dropped); the totals are complete.On
mainf1af6c5 with the scx_lavd scheduler, in two batches:mainagainst 95e7314 (the head before the seal, with two builds of other PRs in the same rotation), then 95e7314 against this head.mainf1af6c5, scx_lavdmainAll builds are slower on f1af6c5 than on b51c65d here,
mainincluded. That comes from #6632's hand-off spin on this machine, not from this change: in alternating runs,mainf1af6c5 took 201.0 ms with the spin and 165.8 ms withAPS5_HANDOFF_SPIN_US=0, andmainb51c65d 167.2 ms.Earlier runs, from before the last rebase: this change at 64dd0ed1d on
mainb51c65d with the scx_lavd scheduler, and at 8250eec36 onmain864729d with the EEVDF scheduler. 95e7314 differs from 64dd0ed1d only by the rebase and by keeping the indirect argument ranges as reads.mainb51c65d, scx_lavdmainmain864729d, EEVDFmainLeft out: runs that hit the #5372 import race (about 1 fps for the whole window: one run of 95e7314 and one of this head on f1af6c5, one
mainrun on b51c65d, and one run of each build on 864729d), and the warm-up runs (on f1af6c5main188.7 ms / 92.5 ms, 95e7314 175.3 / 62.0 and this head 214.1 / 70.6;main157.4 / 89.5 and this change 165.1 / 64.8 on b51c65d; 164.0 / 115.1 and 152.5 / 69.5 on 864729d). Stray is CPU-bound in this scene (queue worker 0 is 97 to 98 % busy in every run), so the GPU busy time per present falls by about a third while the frame time falls by 1 to 3 %, which is close to the spread between runs.Render passes, from one more run of each with
APS5_PROFILE_DRAW=1on b51c65d (mean of the last 8 windows of 10 s):mainbegan a pass for 76,058 of 224,986 draws per 10 s (33.8 %), this change for 17,200 of 228,238 (7.5 %). On f1af6c5 (mean of the windows in the scene):main66,978 of 197,656 (33.9 %), 95e7314 15,990 of 212,286 (7.5 %), this head 15,688 of 206,704 (7.6 %). In this title the seal begins no extra pass: a build with counters (not part of this PR) found no write-only stores in the open pass at any of the about 54,000 seals per 10 s in the scene.Frame dumps (
APS5_DUMP_FRAMES=40 APS5_DUMP_FRAMES_EVERY=104, 960x540, 30 frames per run) from one run ofmainb51c65d, one of this change (64dd0ed1d) and a second one ofmain: this change and the secondmainrun differ by 0.15 per 8-bit channel on average over frames 5 to 29, and by at most 0.86 in one frame, spread over edges and lights like TAA noise. The firstmainrun reached the scene earlier, so its frames differ from both of the others by much more.Traces, summaries and digests: https://github.com/teepean/AnyPS5/releases/tag/perf-write-only-passes-20261018 (each run's trace,
aps5-summary.jsonlanddigest.md, the[draws]lines of the profiled runs, and frame 28 of the dumps with its difference image; the f1af6c5 runs are insummaries-f1af.zip). Thetools/trace_digest.pyoutput, with its summary line, for the first scored run ofmainf1af6c5 and for d92108270 in the second batch (the traces areaps5-trace-f1af6c591588-1386863.json.gzandaps5-trace-d92108270794-1494022.json.gzon the release):`main` f1af6c5 (run xm1)
seconds 185:20 from present 2772: 105 presents, frame p50 205.5 / p90 220.7 / p99 247.1 / max 255.6 ms, GPU busy 91.9 ms per present
NVIDIA GeForce RTX 3090, NVIDIA 615.78.08, Linux 7.2.10-1-cachyos, 3840x2160, commit f1af6c5, dropped: yes
{"commit":"f1af6c591588","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"185:20"},"gpu":"NVIDIA GeForce RTX 3090","driver":"NVIDIA 615.78.08","os":"Linux 7.2.10-1-cachyos","resolution":"3840x2160","window":{"spec":"seconds 185:20","first_present":2772,"ms":20008.453},"presents":105,"frame_ms":{"p50":205.544,"p90":220.71,"p99":247.085,"max":255.632},"gpu_busy_ms_per_present":91.86,"worker_busy_pct":{"queue worker 0x0":97.939,"queue worker 0x20":98.998},"top_phases_ms_per_frame":{"Driver.Packet":374.453,"Driver.Packet.memory_wait":188.129,"Driver.Packet.draw":131.618,"Driver.Draw":130.811,"Driver.Draw.shader_compile_and_link":79.509,"Shader.DrawStage":69.18,"Shader.DrawStage.materialize":38.748,"Driver.Draw.draw_enqueued":36.661},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":4.136,"events_lost":1001669,"totals_full":false}},"trace":"aps5-trace-f1af6c591588-1386863.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 218.3, Driver.Packet.memory_wait 188.1, batch 91.8, Driver.Packet.draw 24.4, Driver.Draw 24.3, Driver.Draw.shader_compile_and_link 12.0, Shader.DrawStage 7.2, Driver.Present 4.8, VideoOut.Flip 4.8, VideoOut.Flip.present 4.8
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
d92108270, this change before the rebase (run ss1)
seconds 185:20 from present 3322: 102 presents, frame p50 205.2 / p90 233.6 / p99 261.4 / max 272.3 ms, GPU busy 68.5 ms per present
NVIDIA GeForce RTX 3090, NVIDIA 615.78.08, Linux 7.2.10-1-cachyos, 3840x2160, commit d92108270794, dropped: yes
{"commit":"d92108270794","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"185:20"},"gpu":"NVIDIA GeForce RTX 3090","driver":"NVIDIA 615.78.08","os":"Linux 7.2.10-1-cachyos","resolution":"3840x2160","window":{"spec":"seconds 185:20","first_present":3322,"ms":20041.866},"presents":102,"frame_ms":{"p50":205.168,"p90":233.637,"p99":261.432,"max":272.275},"gpu_busy_ms_per_present":68.535,"worker_busy_pct":{"queue worker 0x0":98.017,"queue worker 0x20":99.137},"top_phases_ms_per_frame":{"Driver.Packet":386.518,"Driver.Packet.memory_wait":193.758,"Driver.Packet.draw":129.48,"Driver.Draw":128.672,"Driver.Draw.shader_compile_and_link":82.301,"Shader.DrawStage":71.904,"Shader.DrawStage.materialize":40.375,"Driver.Draw.draw_enqueued":32.249},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":4.285,"events_lost":950241,"totals_full":false}},"trace":"aps5-trace-d92108270794-1494022.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 226.4, Driver.Packet.memory_wait 193.7, batch 68.4, Driver.Packet.draw 24.3, Driver.Draw 24.1, Driver.Draw.shader_compile_and_link 12.5, Shader.DrawStage 7.5, Driver.Present 5.8, VideoOut.Flip 5.8, VideoOut.Flip.present 5.7
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code