self.attn ValueError: not enough values to unpack (expected 2, got 1)

[tracyyy-zhu]

Describe the bug

I have encountered an error when trying to sample from SD3 transformer component, where line 208 in attention.py is asking for two outputs but only one is given:

  File "/home/p2p/pytorch/mc3/envs/marigold/lib/python3.10/site-packages/diffusers/src/diffusers/models/attention.py", line 208, in forward
    attn_output, context_attn_output = self.attn(
ValueError: not enough values to unpack (expected 2, got 1)

Reproduction

https://colab.research.google.com/drive/1CkgjIaaClKUk4ZC-g_RR578BfFE24gps#scrollTo=1xcDHPHd56WH

import logging
from diffusers.models.transformers import SD3Transformer2DModel
import torch
from typing import Optional
from torch.nn import Conv2d
from torch.nn.parameter import Parameter

device = torch.device("cuda")
cat_latents = torch.randn(1, 16, 128, 128).to(device)
timesteps = torch.tensor([453.9749]).to(device)
prompt_embeds = torch.randn(1, 154, 4096).to(device)
pooled_prompt_embeds = torch.randn(1, 2048).to(device)

model = SD3Transformer2DModel().to(device)
model.enable_xformers_memory_efficient_attention()

model_pred = model(
    hidden_states=cat_latents,
    timestep=timesteps,
    encoder_hidden_states=prompt_embeds,
    pooled_projections=pooled_prompt_embeds,
    return_dict=False,
)[0]

Logs

Traceback (most recent call last):
  File "/home/p2p/src/trainer/trainer.py", line 347, in train
    model_pred = self.model.transformer(
  File "/home/p2p/pytorch/mc3/envs/marigold/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/p2p/pytorch/mc3/envs/marigold/lib/python3.10/site-packages/diffusers/src/diffusers/models/transformers/transformer_sd3.py", line 347, in forward
    encoder_hidden_states, hidden_states = block(
  File "/home/p2p/pytorch/mc3/envs/marigold/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/p2p/pytorch/mc3/envs/marigold/lib/python3.10/site-packages/diffusers/src/diffusers/models/attention.py", line 208, in forward
    attn_output, context_attn_output = self.attn(
ValueError: not enough values to unpack (expected 2, got 1)

System Info

• 🤗 Diffusers version: 0.32.0.dev0
• Platform: Linux-6.1.0-26-amd64-x86_64-with-glibc2.36
• Running on Google Colab?: No
• Python version: 3.10.12
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.26.2
• Transformers version: 4.46.1
• Accelerate version: 0.34.2
• PEFT version: 0.13.2
• Bitsandbytes version: not installed
• Safetensors version: 0.4.3
• xFormers version: 0.0.21
• Accelerator: NVIDIA RTX A6000, 49140 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza @yiyixuxu @sayakpaul

[sayakpaul]

This is not a reproducible code snippet. Could you please adjust it accordingly? We don't know any values here:

model_pred = self.model.transformer(
    hidden_states=cat_latents,
    timestep=timesteps,
    encoder_hidden_states=prompt_embeds,
    pooled_projections=pooled_prompt_embeds,
    return_dict=False,
)[0]

[tracyyy-zhu]

The inputs are tensors of shape as following:

`cat_latents` torch.Size([1, 32, 128, 128])
timesteps torch.Size([1])
prompt_embeds torch.Size([1, 154, 4096])
pooled_prompt_embeds torch.Size([1, 2048])

[sayakpaul]

I kindly ask you to modify your reproducer to a fully reproducible and minimal one.

[tracyyy-zhu]

Hi @sayakpaul I have modified to include the tensor values.

[sayakpaul]

Your reproducer throws errors. Check the Colab:
https://colab.research.google.com/gist/sayakpaul/289cc8d879b2bfb846c7a95e2eaa7b47/scratchpad.ipynb

[Jamie-Cheung]

Hi @sayakpaul I have modified to include the tensor values.

I encountered the same mistake as you, can you share how you solved it

[tracyyy-zhu]

@Jamie-Cheung from my experience, commenting out this line in code solves the error:
# model.transformer.enable_xformers_memory_efficient_attention()
Fyi I was using diffusers version 0.31.0 in my code base for some compatibility reason. In the following notebook, changing to diffusers==0.32.0 also solves the error.

@sayakpaul This notebook reproduces the error: https://colab.research.google.com/drive/1CkgjIaaClKUk4ZC-g_RR578BfFE24gps#scrollTo=HU4WaNDM9vJk
From my debug, enable_xformers_memory_efficient_attention() changes some attention professors to diffusers.models.attention_processor.XFormersAttnProcessor object at 0x7f8e6581fd60, which returns only one value.

I added a print statement print("self processor", self.processor) before this line: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py#L588
This is what I get when enable_xformers_memory_efficient_attention() is not commented:

self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f8e659a23e0>
self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f8e659a23e0>
self processor <diffusers.models.attention_processor.XFormersAttnProcessor object at 0x7f8e6581fd60>

When I comment it I get (for one forward pass):

self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f45f3646c50>
self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f45f3646c50>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3623070>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3623c70>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3644670>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36450c0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3645990>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3646290>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3646dd0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3647670>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e80a0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e89a0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e9240>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e9e40>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36ea710>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36eb100>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36eba90>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fc310>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fce20>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fd750>

Please feel free to leave any advice or thoughts

[Jamie-Cheung]

@Jamie-Cheung from my experience, commenting out this line in code solves the error: # model.transformer.enable_xformers_memory_efficient_attention() Fyi I was using diffusers version 0.31.0 in my code base for some compatibility reason. In the following notebook, changing to diffusers==0.32.0 also solves the error.

@sayakpaul This notebook reproduces the error: https://colab.research.google.com/drive/1CkgjIaaClKUk4ZC-g_RR578BfFE24gps#scrollTo=HU4WaNDM9vJk From my debug, enable_xformers_memory_efficient_attention() changes some attention professors to diffusers.models.attention_processor.XFormersAttnProcessor object at 0x7f8e6581fd60, which returns only one value.

I added a print statement print("self processor", self.processor) before this line: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py#L588 This is what I get when enable_xformers_memory_efficient_attention() is not commented:

self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f8e659a23e0>
self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f8e659a23e0>
self processor <diffusers.models.attention_processor.XFormersAttnProcessor object at 0x7f8e6581fd60>

When I comment it I get (for one forward pass):

self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f45f3646c50>
self processor <diffusers.models.attention_processor.AttnProcessor2_0 object at 0x7f45f3646c50>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3623070>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3623c70>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3644670>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36450c0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3645990>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3646290>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3646dd0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f3647670>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e80a0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e89a0>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e9240>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36e9e40>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36ea710>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36eb100>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f36eba90>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fc310>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fce20>
self processor <diffusers.models.attention_processor.JointAttnProcessor2_0 object at 0x7f45f35fd750>

Please feel free to leave any advice or thoughts

If I want to use xformers to further reduce GPU memory use. How to solve "diffusers.models.attention_processor.XFormersAttnProcessor object at 0x7f8e6581fd60 only returns only one value"?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[tsvs]

getting the same problem, is there a way to run flux with xformers in diffusers?

[yiyixuxu]

hey! I cannot reproduce the error

are you on the latest diffusers?

can you print out model.attn_processors?

[tsvs]

model.attn_processors: self.attn.processor=<diffusers.models.attention_processor.XFormersAttnProcessor object at 0x722a2a1b3a10

I use: diffusers 0.33.1