[REQUEST] Support Offload deepspeed engine in RLHF training

[hijkzzz]

See vLLM sleep mode: vllm-project/vllm#11743
This feature is very important in RLHF
such as the hybrid engine mode in OpenRLHF: OpenRLHF/OpenRLHF@c10f02b

[tjruwase]

@hijkzzz can you be more specific on this request. DeepSpeed is probably the first framework to offer offloading in RLHF training:
https://github.com/deepspeedai/DeepSpeedExamples/blob/8075143d922e0a25c8217ed4f72ef7121cad423a/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py#L237-L251

[hijkzzz]

@hijkzzz can you be more specific on this request. DeepSpeed is probably the first framework to offer offloading in RLHF training: https://github.com/deepspeedai/DeepSpeedExamples/blob/8075143d922e0a25c8217ed4f72ef7121cad423a/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py#L237-L251

RLHF training needs:
offload model/optimizer to CPU when using vLLM samples generation for larger inference batch size
load model/optimizer back to GPU during training to avoid PCI-E data exchange
same as vllm-project/vllm#11743

[tjruwase]

@hijkzzz, it seems you are requesting on-demand and fine-grained offloading, is that correct? Would the following APIs work:
https://deepspeed.readthedocs.io/en/latest/zero3.html#offload-states

@tohtana, FYI