perf(agc): pipeline DMA_DATA when APS5_PIPELINE_DMA=1

[ivica00]

What

DMA_DATA on queue 0 joins the draw pipeline when APS5_PIPELINE_DMA=1. keepsDrawPipeline uses that same switch: with it unset, every DMA_DATA drains, including a non-overlapping copy and a DST_SEL nowhere prefetch. A copy whose source or destination overlaps an uncommitted draw drains first. A copy that does not, a SET_PREDICATION, and a NOP from the allow-list (plain NOP, dispatch reset, marker push and pop, context state) leave the queue in place. Other custom NOP packets still drain.

SET_PREDICATION (opcode 0x20) only stores the predicate address and operation on the queue. It does not read that memory. The later predicated packet drains when the predicate overlaps a queued write.

A copy reads its source when the pipeline runs it, so it sees an earlier fill. An immediate fill above 64 KiB is still stored as a buffer fill, a copy into host memory is still counted, and a predication that reads an uncommitted range waits for that range. Marker push and pop still update the marker stack.

Tested

Not re-run in this update. The previous run of agc_driver_pipelined_dma_tests and agc_driver_tests was on an AMD Radeon RX 9070 XT, Windows 10. The DMA test sets APS5_PIPELINE_DMA=1 and covers fill-then-copy ordering, a prefetch (DST_SEL nowhere), a zero-byte copy, a GDS copy, an unaligned copy off a pending write, onto a pending write, and of a pending write, SET_PREDICATION, marker push/pop, a predicated packet whose predicate overlaps a pending write, and DISPATCH_RESET staying on the pipeline.

No new title-session numbers from this machine. The measurements already on this PR (Borderlands 4, Stellar Blade) do not show a frame-time win for copies and fills on by default, so pipelining stays opt-in. @Pacuka measured The Smurfs - Dreams with APS5_PIPELINE_DMA=1 (about 7.9 to 9.4 fps in the village). @TheAggie measured the prefetch case on RetroArch's PCSX2 core; that part is #6538, which is still open, so the nowhere case stays here until that lands.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• GPUs tested: AMD Radeon RX 9070 XT, Windows 10
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6946	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp L26
PR 6947	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp L26
PR 7153	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp L35

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4448	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 5115	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 5925	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 6188	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 6292	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 6501	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
PR 6538	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 6615	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp
PR 7464	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp

[adriadam10]

Built this on its own commit (the stack below it is not merged; #6203, its base, was closed as a duplicate of #1201) and ran the libSceAgcDriver tests:

• agc_driver_pipelined_dma (added here) fails: "the copy did not see the fill before it, or wrote past its bytes". Please make it pass and check it fails without the enqueueDmaPacket change.
• agc_driver_pm4 segfaults: the PR drops the marker push/pop in Pm4.cpp:986-991, but tests/Pm4.cpp:220 still checks state.markers.back() == "abc" and the unbalanced-pop case at :223. Either keep the marker stack behaviour or update those tests and say why nothing reads queue.markers.
• Turning the DMA pipeline on by default (PacketMemory.cpp, APS5_NO_PIPELINE_DMA) is a perf change with no numbers. Please post before/after (same scene, several runs) and a Linux test that covers the overlap cases of keepsDrawPipeline case 0x50 (copy overlapping an uncommitted draw write, dst == 1/2, 0 bytes) and the predication wait.
• The PR conflicts with main and stacks on #6203, which was closed as a duplicate of #1201: please rebase it on #1201 (or on main once #1201 is in) and it can be re-reviewed then.

[adriadam10]

Heads-up from testing #1202 on current main (#1201 merged): with APS5_PIPELINE_DMA=1, agc_driver fails at tests/Submit.cpp:537 ("a copy into host memory drained instead of syncing its source"). The bytes are right; the copy is enqueued on the pipeline, so copiesToHostMemory never counts it. With DMA on by default here, that check (or the enqueue path's count) needs updating in this PR.

[TheAggie]

@ivica00 one data point for the keepsDrawPipeline case 0x50 part of this, from a title where it is almost the whole cost: RetroArch's PCSX2 core (Ratchet & Clank) under RADV issues a DMA_DATA prefetch (DST_SEL nowhere) per draw, about 120,000 per 10 s, and with pipelined draws every one drained the pipeline. Keeping the pipeline for that destination alone, with copies and fills still draining, took the heavy intro scenes from 24-25% of real time to 37-43% here over three runs, and the lighter scenes to real time.

Since this PR is being reworked onto #1201 and that part needs no snapshotting, I have opened it on its own as #6538, with a test that holds the pipeline busy and counts the drains for a prefetch and for a copy. If you would rather fold it in here, say so and I will close mine.

AI-assisted (Claude Code)

[ivica00]

Rebased onto current main (#1201 is merged; the closed #6203 stack is no longer the base).

• agc_driver_pipelined_dma failed because a copy of at most 64 KiB snapshotted its source while being enqueued, before the fill ahead of it had run. The copy now reads the source when the pipeline runs it, in order. The same test passes here, and it is what fails if that read moves back to enqueue time.
• Marker push and pop stay. keepsDrawPipeline already leaves those NOPs on the pipeline; dropping the stack made agc_driver_pm4 call markers.back() on an empty vector. DISPATCH_RESET still drains.
• A pipelined copy into host memory now counts copiesToHostMemory on the same path-2 condition as the synchronous path, and an immediate fill above 64 KiB is still a buffer fill (dmaFills). agc_driver_tests passes with DMA on by default.
• The DMA test also covers destination nowhere, destination GDS, a zero-byte copy, an unaligned copy that overlaps an uncommitted write and one that does not, SET_PREDICATION, and a predicated packet whose predicate overlaps an uncommitted write.
• I do not have a new title-session timing run. @TheAggie's PCSX2 prefetch numbers are the measurement for destination nowhere; they are in the description.

Locally, on an RX 9070 XT: agc_driver_pipelined_dma_tests and agc_driver_tests passed.

[milescoviello]

A data point from a title where DMA_DATA drains are frequent: Borderlands 4 (PPSA01494). Linux 6.8, RTX 4070 Ti (595.99.02), Release, pad input, warm cache, 130 s runs. main ea5c27b against the same main with this PR's head (4a2b7d34f2) merged (clean merge).

Profiled runs (APS5_PROFILE_DRAW=1), the last [draw] pipelined line:

Build	packet drains per 10 s	worker time in them	by opcode
main	2,750	4,491 ms	0x3c 1,017, 0x50 763, 0x10/0x20/flip 255 each
this PR	714	5,641 ms	0x3c 358, 0x10/0x3f/flip 90 each, 0x15 87

The DMA_DATA drains are gone, and the count of WAIT_REG_MEM drains also falls by about two thirds. The worker's total time in the remaining drains doesn't fall, though: each one now waits longer.

Frame rate in the 100–110 s window, no APS5_PROFILE_* switch, two rounds interleaved:

main 1: **seconds 100:10** from present 2001: 106 presents, frame p50 20.6 / p90 348.2 / p99 654.5 / max 756.3 ms, GPU busy 7.8 ms per present
PR 1:   **seconds 100:10** from present 2043: 82 presents, frame p50 18.0 / p90 546.5 / p99 954.2 / max 954.2 ms, GPU busy 8.1 ms per present
main 2: **seconds 100:10** from present 2093: 39 presents, frame p50 17.1 / p90 689.8 / p99 3992.6 / max 3992.6 ms, GPU busy 10.7 ms per present
PR 2:   **seconds 100:10** from present 1997: 70 presents, frame p50 19.2 / p90 621.6 / p99 836.4 / max 836.4 ms, GPU busy 5.6 ms per present

That scene hitches hard either way (p90 350–690 ms), so these windows don't show a difference. All 6 runs (3 per build) reached their limits.

AI-assisted: yes (Claude Code)

[adriadam10]

Thanks for the rebase; the points from the earlier review are addressed in the diff (copy reads its source when the pipeline runs it, marker stack kept, copiesToHostMemory counted, overlap/GDS/zero-byte/predication tests). Remaining:

• The only measured gain for this change is the DST_SEL nowhere prefetch (@TheAggie, 24-25% → 37-43% on a PCSX2 scene), and that case is already in #6538 (keepsDrawPipeline case 0x50, same file). Please rebase on top of it once merged and drop the duplicate part.
• For the rest (copies and fills on the pipeline by default) the data in this thread shows no gain: on Borderlands 4 (PPSA01494, Linux, RTX 4070 Ti) the 0x50 drains disappear but the worker's time in drains goes 4,491 → 5,641 ms and frame times are unchanged. Turning a default on needs a before/after that shows a win on at least one title; please post one, or keep it behind APS5_PIPELINE_DMA=1.
• PacketExecution.cpp:46 changes the NOP rule from an allow-list (0, 0x09, 0x0b, 0x0c, 0x1a) to "everything except 0x09". Please say which other NOP operations exist and why none needs the pipeline drained (or keep the allow-list plus 0x0b/0x0c).

[moong15]

Tested this on Windows (RX 9070 XT, AMD 26.8.1) with Stellar Blade, which issues a lot of DMA_DATA on queue 0, merged on top of current main together with other draw-pipeline work. Measured with APS5_PROFILE_DRAW=1 (per 10 s intervals, same scenes, before = this PR not applied, after = this PR's default):

	before	after
DMA_DATA (0x50) packet drains, menu	9,500-9,900	0
DMA_DATA packet drains, gameplay	2,300-5,600	0
All queue-0 packet drains, menu	~20,000 (2.3-2.5 s)	~13,000 (1.8 s)

No pipeline failures or crashes across three ~10 minute runs with it on by default, and nothing looked wrong while playing. agc_driver_pipelined_dma passes on the merged tree.

Frame time in Stellar Blade did not move much, because the queue-0 worker now waits on the draw committer at other packets instead, so this does not show up as FPS here yet. The drain removal itself works as described.

🤖 Generated with Claude Code

[moong15]

A possible ordering gap in keepsDrawPipeline for DMA_DATA (PacketExecution.cpp, the case 0x50 this PR adds). It decides whether a DMA_DATA that stays on the inline path needs a drain by checking only the destination against queued writes:

const auto address = packet[4] | (static_cast<std::uint64_t>(packet[5]) << 32u);
return !DrawPipeline::Queue0().Overlaps(address, bytes);

The source is never checked. A memory-to-memory copy that enqueueDmaPacket refuses (for example an unaligned destination or size) therefore runs inline without a drain, even when a queued item still has to write its source. The copy then reads the source before that item commits and stores stale bytes.

I hit it while working on a change that keeps some small copies off the pipeline: a fill followed by a copy of the filled range read zeros. Checking the source as well fixed it:

const auto source = Pm4::DmaSource(packet);
if ((source == 0 || source == 3) && DrawPipeline::Queue0().Overlaps(packet[2] | (static_cast<std::uint64_t>(packet[3]) << 32u), bytes)) return false;
return !DrawPipeline::Queue0().Overlaps(address, bytes);

I haven't reproduced it on this PR's head alone, only on a branch that sends more copies down the inline path. On this PR's head an unaligned copy of a range a queued fill writes should show it. I can write that test if it helps.

[ivica00]

Addressing the remaining review:

• Pipelined copies and fills are opt-in again (APS5_PIPELINE_DMA=1). The Borderlands 4 and Stellar Blade numbers on this PR do not show a frame-time win, and the PCSX2 gain was the DST_SEL nowhere prefetch. That part is perf(agc): a DMA_DATA prefetch keeps the draw pipeline #6538, which is still open, so it stays here until that lands and can be dropped.
• Opcode 0x10 is the allow-list again: plain NOP, DISPATCH_RESET (0x09), marker push/pop (0x0b/0x0c) and context state (0x1a). The other custom operations (DRAW_RESET, WAIT_FLIP_DONE, ACQUIRE_MEM, WRITE_DATA, FLIP, RELEASE_MEM, DMA_DATA_CUSTOM) are not NOPs and still drain.
• keepsDrawPipeline for DMA_DATA also drains when a memory source (selector 0 or 3) overlaps a queued write. An unaligned copy that stays on the worker then cannot read that range early. The DMA test covers a 2-byte copy of a range a queued item still writes.

Pushed in bf47f84.

[oneandonlydean]

Picking this up while @adriadam10 is away. Thanks: the enqueue is opt-in now and the NOP allow-list is back. Two things remain. (1) keepsDrawPipeline's new case 0x50 still applies without APS5_PIPELINE_DMA, so by default non-overlapping DMA_DATA no longer drains. Please gate it on the same switch, or post a before/after showing a win. Please also say why opcode 0x20 joins the keep list. (2) Once #6538 merges, please rebase on it and drop the DST_SEL-nowhere case it already covers.

[Pacuka]

Data point where this helps: The Smurfs - Dreams (PPSA21607, Unreal Engine), Linux, RX 6650 XT / RADV Mesa 26.2.4, main 9f25d0b + this PR (plus local unrelated rendering fixes). In the Smurfs' Village, APS5_PROFILE_DRAW=1 shows ~10k DMA_DATA per 10 s on queue 0 taking ~3.8 s (each a drain). With APS5_PIPELINE_DMA=1: 76–84 → 92–96 presents per 10 s (~7.9 → ~9.4 fps, about +19%), steady over ~15 windows; DMA_DATA drops out of the top packets. Time shifts partly into fill HLE dispatches (65 ms → 1.1 s per 10 s) and packet drains (1.2 s → 4.2 s), but the net is a clear win here. No rendering differences noticed.

AI-assisted (Claude Code).

[ivica00]

Addressed on 2d96ff77a.

1. keepsDrawPipeline for DMA_DATA (0x50) now uses the same APS5_PIPELINE_DMA check as enqueueDmaPacket. With the switch unset, every DMA_DATA drains, including a non-overlapping copy and a DST_SEL nowhere prefetch. @Pacuka's Smurfs numbers are with the switch on, so they stay the measurement for the opt-in path. The default path is a drain again.

Opcode 0x20 is SET_PREDICATION. It only stores the predicate address and operation on the queue; it does not read that memory. The packet that is actually predicated still drains when the predicate overlaps a queued write. The DMA test covers that with opcode 0x13, and it expects 0x20 itself not to drain. That is why 0x20 is on the keep list.

2. perf(agc): a DMA_DATA prefetch keeps the draw pipeline #6538 is still open, so the DST_SEL nowhere case stays in this case 0x50 until that merges. I will rebase onto it and drop the duplicate then.

The DMA test sets APS5_PIPELINE_DMA=1 before it counts drains, so this gate does not change that run. I did not re-run it on a GPU for this push.

The NOP allow-list and opt-in gating are done; the #6538 overlap now only affects merge order.

[oneandonlydean]

Merged while @adriadam10 is away. Of your three points, the NOP allow-list and the opt-in gating are done, and the overlap with #6538 now sits inside the opt-in path, so whichever of the two lands second resolves one small conflict. Full CI is green on this head.