[Feat] Adding minimal training for multimodal model

[kcz358]

This PR integrates the training of Qwen2-VL into the open-r1. Due to the implementation of the current GRPO trainer, I have to hack the loading and processing logic of the model inside the trainer and create a trainer class.

The original training code is in :
https://github.com/EvolvingLMMs-Lab/open-r1-multimodal

The logs, checkpoints, and dataset used are available as follows:

Logs : Wandb Logs
Models : 🤗 Models
Datasets : 🤗 Datasets

The runs of the train can be done by torchrun

torchrun --nproc_per_node="8" \
    --nnodes="1" \
    --node_rank="0" \
    --master_addr="127.0.0.1" \
    --master_port="12345" \
    src/open_r1/grpo.py \
    --deepspeed scripts/zero3.json \
    --output_dir checkpoints/Qwen2-VL-2B-GRPO-8k \
    --model_name_or_path Qwen/Qwen2-VL-2B-Instruct \
    --dataset_name lmms-lab/multimodal-open-r1-8k-verified \
    --max_prompt_length 8192 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --logging_steps 1 \
    --bf16 \
    --report_to wandb \
    --gradient_checkpointing true \
    --attn_implementation flash_attention_2 \
    --max_pixels 2359296 \
    --save_total_limit 8 \
    --num_train_epochs 1 \
    --run_name Qwen2-VL-2B-GRPO-8k

[Luodian]

@qgallouedec @lewtun

We truly appreciate this project. Thanks for your efforts!

We are wondering if you could consider adding multimodal models training to the project. It would be exciting to see the multimodal reasoning research derived from this project. We’d love to hear your thoughts on this!