USP method with para_attn of Chengzy in Flux

[TuanNT-ZenAI]

In Para_Attn, Chengzy use hybird Ring and Ulysses, why in this repo use only Ulysses (with 2 GPU)?

[feifeibear]

The USP (Unified Sequence Parallelism) approach in xDiT is flexible, allowing you to configure it as needed.

For example, you can use Ring parallelism on 2 GPUs by setting ulysses_degree=1 and ring_degree=2. This flexibility enables users to adapt the parallelism strategy based on their specific hardware and performance requirements.

Also, we are happy to notice that USP which is first proposed by the team of xDiT has been widely adopted in other repos.