Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置#626
Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置#626Ankur-singh wants to merge 1 commit into
Conversation
Document the validated NVFP4 EAGLE3, DCP, and native CPU KV offload paths from InferenceX PR #2158. 中文：补充 InferenceX PR #2158 已验证的 Kimi-K2.6 NVFP4 B300 EAGLE3、解码上下文并行和原生 CPU KV 缓存卸载配置。 Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Kimi-K2.6 model configuration to support NVIDIA B300 and Blackwell GPUs, bumping the minimum vLLM version to 0.25.0, pinning the NVIDIA docker image to a nightly build, and introducing a native CPU KV offload feature. The nvfp4 variant is updated with optimized parameters, and the markdown guide is expanded to document these new serving paths. Feedback suggests adding VLLM_USE_SIMPLE_KV_OFFLOAD: "1" to the nvfp4 variant's extra_env so that the environment variable is correctly generated in the UI command when the CPU KV offload feature is enabled.
| extra_env: | ||
| VLLM_USE_FLASHINFER_MOE_FP4: "1" | ||
| VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm" |
There was a problem hiding this comment.
The guide specifies export VLLM_USE_SIMPLE_KV_OFFLOAD=1 for the native CPU KV offload path. However, since features in the command builder do not support contributing environment variables, this environment variable won't be generated in the UI command when the native_cpu_kv_offload feature is enabled. To ensure the generated UI commands work seamlessly, consider adding VLLM_USE_SIMPLE_KV_OFFLOAD: "1" to the nvfp4 variant's extra_env.
extra_env:
VLLM_USE_FLASHINFER_MOE_FP4: "1"
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
VLLM_USE_SIMPLE_KV_OFFLOAD: "1"
ivanium
left a comment
ivanium
left a comment
There was a problem hiding this comment.
Overall looks good and self contained. We can remove the synthetic spec decoding args.
| - "--attention-backend" | ||
| - "TOKENSPEED_MLA" | ||
| - "--speculative-config" | ||
| - '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' |
There was a problem hiding this comment.
We can remove synthetic args:
| - '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' | |
| - '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,}' |
| --max-num-batched-tokens 16384 \ | ||
| --stream-interval 10 \ | ||
| --enable-prefix-caching \ | ||
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' |
There was a problem hiding this comment.
Same here:
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' | |
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}' |
| --max-num-batched-tokens 16384 \ | ||
| --stream-interval 10 \ | ||
| --enable-prefix-caching \ | ||
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' \ |
There was a problem hiding this comment.
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' \ | |
| --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}' \ |
2 participants
Summary
moonshotai/Kimi-K2.6recipe with the B300 NVFP4 configuration validated by Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试 SemiAnalysisAI/InferenceX#2158SimpleCPUOffloadConnectorfeatureValidation
node scripts/build-recipes-api.mjs✓ JSON API: 147 models ... 8 strategies, 2 platformsSimpleCPUOffloadConnectorpublic/nvidia/Kimi-K2.6-NVFP4.jsonis generatedSource
中文说明
moonshotai/Kimi-K2.6recipeSimpleCPUOffloadConnector原生 CPU KV 缓存卸载功能验证
node scripts/build-recipes-api.mjs✓ JSON API: 147 models ... 8 strategies, 2 platformsSimpleCPUOffloadConnectorpublic/nvidia/Kimi-K2.6-NVFP4.jsoncc @faradawn for review