Skip to content

[Sm90] mxfp8#67

Open
KareemMusleh wants to merge 5 commits into
Dao-AILab:mainfrom
KareemMusleh:sm90-fp8-blockscaled
Open

[Sm90] mxfp8#67
KareemMusleh wants to merge 5 commits into
Dao-AILab:mainfrom
KareemMusleh:sm90-fp8-blockscaled

Conversation

@KareemMusleh

Copy link
Copy Markdown

Right only the forward is implemented. Depends on this quack PR

Will write a summary tomorrow

@KareemMusleh

KareemMusleh commented Jul 7, 2026

Copy link
Copy Markdown
Author

The difference between this and bf16 sonicmoe is that we quant the activations before the up_proj/down_proj. And that mxfp8 scales of A need to be dQaccum padded so that we can load them effectively with TMA. In the forward this happens in two places:

  1. Before the up_proj. Group by expert and pad the scales. This is done as a separate kernel gather_padded_sfa.
  2. Before the down_proj. We just pad the scales because they are already grouped by expert. This is fused into the quant kernel. Now it's its own kernel.

We pre-compute the metadata needed to do this effectively (padded_gather_idx and padded_grouped_idx).

I'd be glad to listen to your feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant