Skip to content

Add text-only Llama 4 support to mini-sglang. - #114

Closed
sheepfish5 wants to merge 16 commits into
sgl-project:mainfrom
sheepfish5:add_llama4
Closed

Add text-only Llama 4 support to mini-sglang.#114
sheepfish5 wants to merge 16 commits into
sgl-project:mainfrom
sheepfish5:add_llama4

Conversation

@sheepfish5

Copy link
Copy Markdown

Summary

This PR adds support for Llama 4 to mini-sglang.

Llama 4 currently has two public model variants:

  • Scout: about 109B total parameters
  • Maverick: about 400B total parameters

This PR implements the text-only Llama 4 model path.
Although Llama 4 natively supports image input, multimodal/image support is not included in this PR.

What’s included

  • Added Llama 4 model support in mini-sglang
  • Implemented the text model path for Llama 4

Tested model

I tested this implementation with the Hugging Face model:

  • meta-llama/Llama-4-Scout-17B-16E-Instruct

Runtime / memory usage

Test environment:

  • 4 x 96GB GPUs

Observed memory usage:

  • about 58GB VRAM per GPU on average

Run command:

python3 -m minisgl --model-path "./models" --shell --attention-backend fi --tp 4

Notes

  • This PR is for text inference only
  • Image / multimodal support for Llama 4 is not implemented here
image

@DarkSharpness DarkSharpness added the enhancement New feature or request label May 10, 2026
@sheepfish5 sheepfish5 closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants