Update on the development branch

[Shixiaowei02]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Aug 20, 2024.

This update includes:

• Model Support
• Features
  • Support for mamba2 TP
  • Integration of lookahead decoding layer into IFB
  • Support for returning generation logits in streaming mode
• API
  • [BREAKING CHANGE] Set use_fused_mlp to True by default
  • [BREAKING CHANGE] Add LogitsPostProcessorConfig
  • Add FinishReason to Result
• Bug fixes
  • Fix cross attention remove padding issue (T5 model, large difference in results when remove_input_padding is enabled #1999)
  • Fix the failure in converting qwen2-0.5b-instruct when using smoothquant (convert qwen2-0.5b-instruct failed when using smoothquant #2087)
  • Match the exclude_modules pattern in convert_utils.py to the changes in quantize.py ([Fix] Match exclude_modules pattern in convert_utils.py to quantize.py changes. #2113)
  • Fix build engine error when FORCE_NCCL_ALL_REDUCE_STRATEGY is set
  • Bug fix for VILA example running
  • Fix unexpected truncation in gpt_attention quant mode
  • Fix race condition causing hang when canceling requests
• Memory optimization
• Benchmark
  • Update perf-overview.md
• Performance
• Infra
• Documentation
  • Extend the multimodal example for covering LLAVA in TensorRT-LLM backend

Thanks,
The TensorRT-LLM Engineering Team