Skip to content

Conversation

@farlukas
Copy link

@farlukas farlukas commented Dec 3, 2025

Purpose

Fuse KV projection with torch.cat in DeepSeek R1 0528. torch.cat launches a separate kernel that takes quite some time. We can eliminate this latency by doing the concatenation together with the KV projection in one kernel by using a fused GEMM + split + concat kernel.

This PR depends on ROCm/aiter#1553.

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results

@farlukas farlukas changed the title [TRITON] Add Fused GEMM A8W8 + Split + Concat to Deepseek FP8 Model [Triton] Add Fused GEMM A8W8 + Split + Concat to Deepseek FP8 Model Dec 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants