Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wire in KV cache quantization #77

Merged
merged 3 commits into from
Jan 17, 2025
Merged

Wire in KV cache quantization #77

merged 3 commits into from
Jan 17, 2025

Conversation

neilmehta24
Copy link
Member

@neilmehta24 neilmehta24 commented Jan 16, 2025 •
edited by yagil
Loading

See this PR for the mlx-lm implementation ml-explore/mlx-examples#1075

Summary of changes:

  • Wire kv_bits, kv_group_size, and quantized_kv_start into model_kit, load_model, and create_generator. These values are optional, so that way we rely on mlx-lm to set defaults for us. kv_bits is required to set the other two parameters
  • Disable kv cache quantization for VLMs
  • Note that max_kv_size is not respected when quantizing the KV cache
  • Apply a formatting pass

Closes #31

@neilmehta24 neilmehta24 marked this pull request as ready for review January 16, 2025 23:06
@neilmehta24 neilmehta24 requested review from yagil and mattjcly January 16, 2025 23:06
@neilmehta24 neilmehta24 merged commit 6b679ef into lmstudio-ai:main Jan 17, 2025
@neilmehta24 neilmehta24 deleted the kv-cache-qtn branch January 17, 2025 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattjcly mattjcly mattjcly approved these changes

@yagil yagil Awaiting requested review from yagil

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add KV cache quantization feature
2 participants
@neilmehta24 @mattjcly