How can I fixate a voice for an audiobook?

[WimTip]

Hi Peter,
Thanks for the Parkiet package, the quality and range of voices is impressive.
I want to use this package to create an audiobook from Max Havelaar by Multatuli (no copyright restrictions, also no audiobook on Bol.com).
For this I would like to have control over what voice is used. Can you guide me?
Again thanks for the package, keep up the good work!
Kind regards, Wim

[pevers]

Hi @WimTip , thanks! And cool project.

Generating long sequences is still a bit hard. But it can work if you stitch it carefully together.

In the generate method in inference.py you can add an audio_prompt parameter

parkiet/src/parkiet/dia/inference.py

Line 23 in d85a55b

output = model.generate(

It should be a .wav or .mp3 for example with ~ 10 seconds of audio of a voice (can be an open voice or your own).

Then you can generate part of the text which is currently limited to 30 seconds only. So you would have to batch it in small pieces and then stitch it together.

I'm working now on the next iteration to generate much longer samples with similar quality.

[0xrushi]

Hi @pevers , thanks for sharing this. Do you have any suggestions for cases where the WAV file used in the audio prompt sounds very different from the generated output?
Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

[pevers]

Hi @pevers , thanks for sharing this. Do you have any suggestions for cases where the WAV file used in the audio prompt sounds very different from the generated output? Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

Yes I noticed this as well. It is not always perfect. Longer samples work a bit better but they also fill up the context. Podcast like speech as sample is the best way to get good results.

Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

Yes, you can fine-tune with a specific voice, you don't need much data. The data is synthetically generated via https://github.com/pevers/whisperd-nl . So if you have a couple of hours of voice data you should be able to transcribe it with disfluencies. The emotions are hard and I only implemented (laughs) so far.