-
Notifications
You must be signed in to change notification settings - Fork 541
Open
Description
https://github.com/THUDM/slime/blob/main/slime/backends/sglang_utils/sglang_engine.py#L250
def update_weights_from_tensor(
self,
serialized_named_tensors: list[str],
load_format: str | None = None,
flush_cache: bool = False,
weight_version: str | None = None,
):
"""
Update model weights from tensor data. The HTTP server will only post meta data, and the real weights will be copied directly from GPUs.
Note: The model should be on GPUs rather than CPU for this functionality to work properly.
If you encounter issues, ensure your model is loaded on GPU devices rather than CPU.
"""i walk through update_weights_from_tensor logic, tensor are serialized and deserialized with sglang endpoint, can anyone help explain this comment?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels