40% Less VRAM Usage, Slightly Faster Inference, and Improved Voice Consistency – Optimized Text Autochunking

[RobertAgee]

Title

40% Less VRAM Usage, Slightly Faster Inference, and Improved Voice Consistency – Optimized Text Autochunking

Important

• This PR depends on UI updates from PR Gradio UI Improvements - Added Gradio Seed Input, Audio Prompt Transcript Field, Console Log Outputs #137.
  ➔ Please review and merge Gradio UI Improvements - Added Gradio Seed Input, Audio Prompt Transcript Field, Console Log Outputs #137 first.

Changes

• ➕ Added smart autochunking of user text input based on effective character length (~48–64 chars)
• ➕ Supports Tensor Core acceleration:
  • Autochunking ensures sequence lengths are hardware-aligned (~64 tokens)
  • Batching improves GPU occupancy and compute efficiency
• 🧠 Refactored model.generate() and _prepare_generation():
  • Now accepts audio_prompt_text as a clean, separate parameter
  • Internally joins it with input text to match previous behavior but with cleaner architecture
• ➕ Autocasts generation to float16 (uses torch.cuda.amp.autocast)
• 🧹 Includes internal memory cleanup after generation for more stable long-running usage

Why

• ✅ ~40% less VRAM usage (~4GB vs ~7GB) on T4 GPUs.
• ✅ Slightly faster inference (~0.3 RTF vs ~0.33 RTF) due to smarter batching and TensorCore activation.
• ✅ Improved voice consistency when using audio prompts, even across multiple chunks.
• ✅ Cleaner internal design (separated audio prompt vs user text, easier future upgrades).

Notes

• 🛠 Tested extensively on Gradio 4.x.
• 🔄 Should be fully backward compatible without impacting any existing workflows.
• 📈 Model behavior and output quality are preserved or improved.
• Test it in Google Colab: https://colab.research.google.com/drive/1g5yocOEeW9IB7ZdmdWs3aHnk6-YA8DfB?usp=sharing

---

Thanks for reviewing this PR! 🚀

[znraznra]

This works well in my case.

[buttercrab]

Can you fix the lint & format

[journeytosilius]

I get really weird results when using expressions with symbols like $140 ... is this common and what are the guidelines to get it right ?

[RobertAgee]

Hey @buttercrab , the linting and format should now be fixed. I also went ahead and tested the merge and everything seems to work.

A couple notes from testing the merge:
1. Good news, there's much fewer weird gaps of silence in the generations 🚀

2. Voice consistency still needs to be improved. 🔧

• Currently, each chunk of audio references the uploaded audio+text, with hopes of capturing the voice. I've also tried referencing each previous chunk+text as it generates, but that causes it to gradually morph away from the speaker. It is possible to tune the settings to mitigate this to a small degree but it's not a workable solution. I have a couple other ideas I want to try by inserting speaker tags but I assume simply finetuning a model will be the best and easiest option regardless.

3. Model is much slower now? 🐢

• While we still have preserved the VRAM savings (~4.2GB baseline, 6.9GB peak on a T4), it seems the other PR changes in the last couple days have made the generations much slower, anywhere from 10-30%, especially on longer audios. Specifically I'm talking about the generations after the initial compilation. Prior the rate was consistent for all lengths of generated audio.

[harmlessman]

1.	How long did it take to generate around 10 seconds of audio from text?
2.	Did you implement streaming functionality for the TTS output?

[RobertAgee]

1.	How long did it take to generate around 10 seconds of audio from text?
2.	Did you implement streaming functionality for the TTS output?

@harmlessman Typically PR requests are not the place to ask these questions, just fyi. However, the information may be useful to @buttercrab.

1. Depends on your GPU card and set up. For reference, for 10s audio generation, original Dia repo on T4 took ~35 s on average. The new compile Dia takes ~40 s on average. My original PR took ~30 s on average.

2. Streaming outputs are not currently supported. A much better card(s) are needed to get faster than real time. We could implement buffered outputs, but there'd still be some initial gap between prompt and output, depending on chunk size.

[buttercrab]

Can you fix the lint error?

[V12Hero]

@RobertAgee to get around the link error just use ruff like:
ruff check path/to/file --fix --unsafe-fixes
and
ruff format path/to/file

It should help you get around any formatting errors.