Can it support segmented processing of long audio or audio streaming?

[mqgw]

Compared to generating results for a long audio all at once, segmented processing or splitting long audio into a stream of short audio clips can reduce the time to first frame of video, decreasing waiting time.

[set-soft]

Or perhaps some kind of "tiled decode" (in reference to tiled VAE decode used for images).
The memory is consumed by:

data_out = self.decode_latent_into_image(s_r = s_r, s_r_feats = s_r_feats, r_d = sample)

This is why I mention decoding by segments

[set-soft]

OK, I asked some help to Gemini Pro and we refactored decode_latent_into_image and save_image.

With this I was able to generate a 4 minutes long video using less than 6 GiB of VRAM and less than 8 GiB of RAM (in a machine with 12 GiB of VRAM and 32 GiB of RAM).

The trick is to avoid:

1. Have the full video in VRAM, I'm creating it on RAM
2. Create the whole video to then adapt it to torchvision images, I'm doing it on-the-fly

These two things drastically reduces the amount of VRAM and RAM needed.

[set-soft]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

[nitinmukesh]

@set-soft

Thank you.

I made this change

def decode_latent_into_image

            img_t_cpu = img_t.cpu()
            d_hat.append(img_t_cpu)

[set-soft]

Hi @nitinmukesh !
This will take you a little farther, but you'll need insane amounts of RAM. My fork does much more than this.
I started doing this, but soon found I needed 60 GB of RAM for the 4 minutes video, and as soon as the code tried to convert the images to frames to create the video I ran out of memory.
The code in my fork avoids such a bad memory management by converting the images on-the-fly.

[nitinmukesh]

I noticed it. For shorter audio it works, but for 4 min audio it need insane amount of RAM. I will use your fork.
Thank you for sharing.

[adkkml]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

[set-soft]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

For a ComfyUI node you need this repo: https://github.com/set-soft/ComfyUI-FLOAT/
The https://github.com/deepbrainai-research/float/ is the original implementation of the paper, CLI Python app, not ComfyUI nodes.

[adkkml]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

For a ComfyUI node you need this repo: https://github.com/set-soft/ComfyUI-FLOAT/ The https://github.com/deepbrainai-research/float/ is the original implementation of the paper, CLI Python app, not ComfyUI nodes.

Using the set-soft repo, it wouldn't let me create a long video (OOM on GTX3090 24GB VRAM). If using your FLOAT.py (with edited some line in the top part to be the same as set-soft), it allow to generate long FLOAT video. Thank You. But sometimes, when saving the video the RAM would hit 100% which makes the computer "hang" a bit. If try using your generate.py it ended with error as follows:

File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FLOAT\generate.py", line 188, in save_video_frames
os.makedirs(os.path.dirname(video_path) or '.', exist_ok=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 251, in dirname
File "", line 213, in split
TypeError: expected str, bytes or os.PathLike object, not NoneType

[set-soft]

Hi @adkkml !

1. Use the full code from https://github.com/set-soft/ComfyUI-FLOAT/ update it, is evolving. Use git pull. And avoid mixing it.
2. If the above mentioned code fails please post the full traceback in an issue for https://github.com/set-soft/ComfyUI-FLOAT/ do not ask for help here, this will confuse users. I can generate more than 5 minutes of video with 12 GB of VRAM and 32 GB of RAM (+ 32 GB of swap, virtually 64 GB).
3. If you want a command line version of the code try: https://github.com/set-soft/float

Both are forks that I modified to allow long videos using an RTX3060

P.S. The command line version needs less memory than the ComfyUI nodes.
P.S.2. Don't confuse the https://github.com/yuvraj108c/ComfyUI-FLOAT nodes, these use almost the same code found in this repo

[adkkml]

Hi @adkkml !

1. Use the full code from https://github.com/set-soft/ComfyUI-FLOAT/ update it, is evolving. Use git pull. And avoid mixing it.
2. If the above mentioned code fails please post the full traceback in an issue for https://github.com/set-soft/ComfyUI-FLOAT/ do not ask for help here, this will confuse users. I can generate more than 5 minutes of video with 12 GB of VRAM and 32 GB of RAM (+ 32 GB of swap, virtually 64 GB).
3. If you want a command line version of the code try: https://github.com/set-soft/float

Both are forks that I modified to allow long videos using an RTX3060

P.S. The command line version needs less memory than the ComfyUI nodes. P.S.2. Don't confuse the https://github.com/yuvraj108c/ComfyUI-FLOAT nodes, these use almost the same code found in this repo

Thank you for the guidance. I tried set-soft/ComfyUI-FLOAT, got OOM (3.09 minutes audio). I will post the error in the set-soft/ComfyUI-FLOAT.

[cleverpal-ai]

Did you finally manage to make generation play out faster by injecting streaming audio? If the audio is rather long, it would be good to start video play while the rest of it generates. Any idea how to realize?

[set-soft]

Hi @SuperMaximus1984 !

You don't really need streaming to support long clips, and FLOAT is quite fast.

I understand your question is oriented to streaming, which isn't currently possible.

BTW: The very advanced nodes (From FLOAT_Optimized) does a memory efficient processing