Skip to content

[Usage] does update_weights_from_tensor save weights data ? #1536

@xgwang

Description

@xgwang

https://github.com/THUDM/slime/blob/main/slime/backends/sglang_utils/sglang_engine.py#L250

    def update_weights_from_tensor(
        self,
        serialized_named_tensors: list[str],
        load_format: str | None = None,
        flush_cache: bool = False,
        weight_version: str | None = None,
    ):
        """
        Update model weights from tensor data. The HTTP server will only post meta data, and the real weights will be copied directly from GPUs.

        Note: The model should be on GPUs rather than CPU for this functionality to work properly.
        If you encounter issues, ensure your model is loaded on GPU devices rather than CPU.
        """

i walk through update_weights_from_tensor logic, tensor are serialized and deserialized with sglang endpoint, can anyone help explain this comment?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions