Skip to content

metal: GPU-side sync for labels, waits and indirect draws/dispatches (async phase 2) - #517

Merged
RINNECODER merged 2 commits into
mainfrom
perf/async-p2-gpu-sync
Oct 15, 2026
Merged

RINNECODER merged 2 commits into
mainfrom
perf/async-p2-gpu-sync

Conversation

@RINNECODER

@RINNECODER RINNECODER commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Phase 2 of the async GPU roadmap: GPU-side synchronization instead of CPU drains.

What changes

  • DISPATCH_INDIRECT: the GPU reads the group counts from the guest argument buffer (dispatchThreadgroupsWithIndirectBuffer). Before, the worker drained the timeline and read them on the CPU (cause 0x16).
  • DRAW_INDIRECT / DRAW_INDEX_INDIRECT (and multi forms with a packet count): Metal indirect draws read each record from the guest buffer. Index and vertex buffers are bound over their declared ranges.
    • Where no shader reads the CP's base-vertex or first-instance SGPR write (the Constant rule), a blit copies each record's count, instances and first index next to the driver's constants, on the GPU.
    • Fans, strips without restart, mesh, rect lists and barycentric capture keep the CPU path (cause 0x10f).
  • Label points (RELEASE_MEM / WRITE_DATA behind committed work) are signalled on a MTLSharedEvent the timeline owns (MetalTimeline::DeferSignal), so GPU work on another queue can wait for them (Phase 4).
    • A label with an end-of-pipe interrupt, or one behind a full batch, commits the batch there, so it is not stored behind work encoded after it.
    • Committing at every label doubled the command buffers per frame (140 to 270) and the worker's encode time, so other labels ride their batch.
  • WAIT_REG_MEM: pending label stores are tracked until they run, not until any completion ran. A wait on a value no pending store of ours provides now polls for it while the timeline retires work, instead of draining first.
  • ANYPS5_METAL_GPU_INDIRECT=0 and ANYPS5_METAL_GPU_WAITS=0 restore the CPU paths.
  • Not touched: the index-scan and DCC drains (0x102, Phase 1). No page write-protection.

Drains per live-map replay frame (local drain-site log, not committed)

cause site before after
0x10f replay tool: drain before it applies the captured guest-CPU page deltas (replay only, not in game) 10 10
0x10f indirect draw arguments read on the CPU 3 1 (barycentric capture)
0x16 indirect dispatch arguments read on the CPU 1 0
0x102 index scan, DCC keys (Phase 1) 5 5

So 10 of the 13 "0x10f" waits in the frame profile come from the replay harness, not the driver. The guest-driven drains left in the frame are the one barycentric indirect draw and the Phase 1 reads.

Correctness

Since #514 the live map has no stable hash. Each repeat's output was dumped and compared:

  • Against replay-ref-v2.png, every repeat, for this branch and for an origin/main build of the same base:
    • live map, main: 85, 81, 83, 67, 77, 128 px; branch: 97, 68, 84, 76, 80, 73 px (gate 200).
    • menu, main: 473 (warm-up), 79, 89, 89, 81, 85 px; branch: 442 (warm-up), 77, 61, 89, 75, 76 px. The warm-up repeat is over 200 on main too.
  • Against main's own noise:
    • Branch pixels outside the per-pixel range of 6 main repeats: live 0–3 px per repeat; menu 0–37 px, all in the animated character.
    • Timed-repeat pairs: live main–main median 89 px, main–branch 82 px; menu 54 px and 56 px.
  • Unit tests (metal_utility_tests):
    • LabelEvents.mm: a consumer on a second queue, committed before its producer, waits for the label event on the GPU and sees the producer's writes with no CPU drain. A polled label is stored while work encoded after it is not even committed. Also covers a label behind committed work and a label riding its batch.
    • ShaderPipeline.mm: an indirect dispatch runs with group counts earlier GPU work wrote; zero groups dispatch nothing.
    • Both were checked to fail when the batch commit at the label, or the indirect offset, is broken.

Timing A/B

10 interleaved pairs against an origin/main build of the same base, live map, under the GPU lock. Traced builds (frame-trace instrumentation, not committed); medians of repeats 2-3 of each run; each variant replays its own warmed clone of the capture.

metric (ms per frame) main this PR paired PR/main PR better
wall 1016.0 1040.2 0.984 6/10
worker blocked on GPU/host 486.9 508.8 0.959 6/10
overlap (GPU busy while worker encodes) 163.3 170.1 1.046 7/10
GPU busy 565.7 570.3 0.992 5/10
worker encode/decode 428.9 430.1 0.975 7/10

The medians move with machine load between pairs (each half drifts), so the paired ratios are the comparison: neutral to about 2-5% better. Command buffers per frame stay at main's 136-139.

  • Why only neutral: the 5 Phase 1 reads (0x102) are still about 90% of the worker's blocked time, and 10 of the 13 0x10f waits belong to the replay tool.
  • What it unlocks for Phase 3: no drains remain at indirect arguments, label points or waits on values our own work does not write. Label points carry a GPU event value that a second queue can wait on (Phase 4).
  • A first version committed the batch at every label point. That doubled the command buffers (about 140 to 270) and the worker encode time (+50%); it is fixed here.
  • No in-game --map run: the A/B is neutral, not positive.

Codex review findings

# Finding Severity Real? Fix
1 WAIT_REG_MEM: flushing instead of draining can pass on stale bytes that pending GPU work is about to overwrite P1 Yes 965b426: drain first when MayHaveWritten (including unknown BDA writes) or a pending label store overlaps the awaited bytes

Final commit re-checked, every repeat: live vs v2 70, 80, 92, 77, 70, 117 px (0-3 px outside main's envelope); menu 484 (warm-up), 68, 51, 92, 94, 94 px (0-26 px outside main's envelope, character only).

🤖 Generated with Claude Code

https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

Phase 2 of the async GPU roadmap: the worker no longer drains the
timeline for indirect arguments, and label points and waits on a value
no longer drain either.

- DISPATCH_INDIRECT: the GPU reads the group counts from the guest
  argument buffer (dispatchThreadgroupsWithIndirectBuffer) instead of
  the CPU reading them after a drain (cause 0x16).
- DRAW_INDIRECT / DRAW_INDEX_INDIRECT and their multi forms with a
  packet count: the Metal indirect draw reads each record from the
  guest buffer, with the index and vertex buffers bound over their
  declared ranges. Where no shader reads the CP's base-vertex or
  first-instance SGPR write (the Constant rule), a blit copies each
  record's count, instances and first index next to the driver's
  constants on the GPU. Fans, strips without restart, mesh, rect lists
  and barycentric capture keep the CPU path (cause 0x10f).
- Label points (RELEASE_MEM / WRITE_DATA behind committed work) are
  signalled on a MTLSharedEvent the timeline owns (DeferSignal), so GPU
  work on another queue can wait for them. A label with an end-of-pipe
  interrupt, or one behind a full batch, commits the batch there, so
  it is not stored behind work encoded after it.
- WAIT_REG_MEM: pending label stores are tracked until they run (not
  until any completion ran), and a wait on a value no pending store of
  ours provides polls for it while the timeline retires work, instead
  of draining first.

ANYPS5_METAL_GPU_INDIRECT=0 and ANYPS5_METAL_GPU_WAITS=0 restore the
CPU paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj
@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 00a80c22-ec98-42fb-aeb7-ee0ba49f7414
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T11:47:32.004790Z 6fa70cd PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fa70cd0b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +969 to +970
if (!GpuWaits()) CompletePriorGpuWorkAndCopyBack(opcode);
else timeline.Flush();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Drain GPU writers before polling an external wait

When deferred GPU work may write the waited range, merely flushing it lets the first Pm4::WaitSatisfied read stale guest memory before that work completes. If the old value already satisfies the comparison but an earlier draw or dispatch subsequently writes a nonmatching value, the loop exits immediately and later commands run without the required wait. Drain when timeline.MayHaveWritten(address, address + bytes) is true (including unknown writes), or otherwise ensure the producer completes before the initial CPU poll.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 965b426: a WAIT_REG_MEM whose awaited bytes committed GPU work may write (MayHaveWritten, including unknown BDA writes) or a pending label store of ours overlaps now drains first, as before. Only a value no pending work of ours writes is polled without a drain.

A WAIT_REG_MEM on bytes that committed GPU work or a pending label store
of ours may still write drains first, as before, so the wait never
passes on the value from before that work (Codex review on #517). Only
a value no pending work of ours writes is polled without draining.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj
@RINNECODER
RINNECODER merged commit 12eff7a into main Oct 15, 2026
1 check passed
@RINNECODER
RINNECODER deleted the perf/async-p2-gpu-sync branch October 15, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant