FSDP2: evaluate/predict on a fresh Trainer raises in Accelerator.prepare

[qgallouedec]

System Info

• transformers 5.17.0.dev0 (main 41f519e)
• accelerate 1.14.0
• torch 2.13.0+cu130
• python 3.13.13
• 1x H100 80GB

Who can help?

@SunMarc

Information

• My own modified scripts

Tasks

• My own task or dataset (give details below)

Reproduction

Calling evaluate or predict on a Trainer that has not trained raises under FSDP2. No context parallelism involved, plain FSDP2 on a single process is enough.

import torch
from torch.utils.data import Dataset

from transformers import AutoModelForCausalLM, Qwen3Config, Trainer, TrainingArguments


class Tokens(Dataset):
    def __init__(self):
        self.ids = torch.randint(0, 1024, (8, 512), generator=torch.Generator().manual_seed(0))

    def __len__(self):
        return len(self.ids)

    def __getitem__(self, index):
        return {"input_ids": self.ids[index], "labels": self.ids[index]}


config = Qwen3Config(
    vocab_size=1024,
    hidden_size=128,
    intermediate_size=256,
    num_hidden_layers=2,
    num_attention_heads=4,
    num_key_value_heads=2,
    attn_implementation="sdpa",
)
model = AutoModelForCausalLM.from_config(config)
args = TrainingArguments(output_dir="out", per_device_eval_batch_size=2, report_to=[])
trainer = Trainer(model=model, args=args)
print(trainer.evaluate(Tokens()))

accelerate launch --num_processes 1 --use_fsdp --fsdp_version 2 \
  --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP repro_fsdp2_eval_only.py

ValueError: When using FSDP2, a model and optimizer must be passed together to `Accelerator.prepare()`
as the optimizer needs to have its parameters modified after the model is converted.

evaluation_loop prepares the model on its own when eval is called without train:

https://github.com/huggingface/transformers/blob/41f519e741/src/transformers/trainer.py#L2824-L2828

and accelerate rejects a model-only prepare() under FSDP2:

https://github.com/huggingface/accelerate/blob/v1.14.0/src/accelerate/accelerator.py#L1518

The branch already picks prepare_model(model, evaluation_mode=True) for the non-FSDP case, so the FSDP2 arm looks like it just needs the same treatment.

Expected behavior

evaluate and predict work on a fresh Trainer under FSDP2, the way they do under DDP.