Qwen2.5 Model Performance Degradation with long Context

[zhangchushu]

When using the model with a context length greater than 32K, I have observed that while the inference speed does show an improvement, the model's capabilities degrade significantly.
I believe this is an important issue that may impact the usability of the Qwen2.5 or other models in scenarios where longer context lengths are required. I would appreciate it if the relevant team or community members could look into this and provide some insights or possible solutions.

[gmlwns2000]

Hi, @zhangchushu ,

That maybe Qwen2.5 is only trained with up to 32 K. So it requires YaRN to extend the RoPE, but we are not carefully handling YaRN yet. HiP attention will work fine without YaRN at this point.

Can you share the server command line you used, if it is possible for us to dig into it?
And it would be helpful if you share your result on evaluations to compare with our reproduction.

There is several performance tuning guide for HiP to improve the accuracy:

• Increase second_stage_k
• Decrease stage_chunk_size of each stages.
• Increase sliding_window_size
• Reduce interval of mask_refresh_interval
• Using dense attention for first few layers during prefill by set HIP_DEBUG_USING_DENSE_PREFILL=1 envvar.
• Using dense attention during decoding by set HIP_DEBUG_FORCE_DENSE_DECODE=1 envvar.
• Using dense attention for last few queries during prefill by set HIP_DEBUG_LAST_DENSE=64 envvar. You can increase the numbers.
• Change indices of dense layers by adjusting dense_layers.

The above changes can be applied on SGLang by passing --hip-attention-args with a JSON-formatted string.

Our current latest branch is feat/delta for hip-attention and feat/qwen25vl for sglang.

Thanks!