[ROCm] Add Kimi-K2.5 AMD AITER MoE env vars#680
Conversation
Add the AITER MoE and fused shared-experts environment variables to the AMD Kimi-K2.5 recipe override. Signed-off-by: haic0 <haichzha@amd.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds the environment variables VLLM_ROCM_USE_AITER_MOE and VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS to the AMD hardware overrides for the Kimi-K2.5 model. Feedback suggests that applying VLLM_ROCM_USE_AITER_MOE globally under AMD hardware overrides may conflict with the --moe-backend flydsl flag configured for the default INT4 model on MI355X, and recommends moving these environment variables to specific configurations or the mxfp4 variant's environment instead.
| VLLM_ROCM_USE_AITER_MOE: "1" | ||
| VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1" |
There was a problem hiding this comment.
Setting VLLM_ROCM_USE_AITER_MOE: "1" globally under hardware_overrides.amd applies it to all AMD platforms and variants. This includes the MI355X running the default INT4 model, which is documented to use --moe-backend flydsl (lines 230-242). In vLLM, having VLLM_ROCM_USE_AITER_MOE=1 in the environment can conflict with or override the --moe-backend flydsl flag, potentially disabling the optimized FlyDSL path or causing runtime errors. To prevent this conflict, these environment variables should be moved to the mxfp4 variant's extra_env or applied only to the specific configurations that require them.
Keep the AITER MoE and fused shared-experts flags on the MXFP4 variant so the default AMD FlyDSL path is not affected. Signed-off-by: haic0 <haichzha@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Summary
VLLM_ROCM_USE_AITER_MOE=1to the AMD Kimi-K2.5 hardware override.VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1to match the benchmark reproduction path documented in PR [ROCm] Document Kimi-K2.5 MXFP4 MI355X benchmark reproduction #673.Test plan
pythonYAML parse and top-level schema-order check formodels/moonshotai/Kimi-K2.5.yamlgit diff --check -- models/moonshotai/Kimi-K2.5.yamlReadLintsonmodels/moonshotai/Kimi-K2.5.yaml