Question about: "why do you use text to train turn detection not audio?"

[phamkhactu]

I'm very appricated for your work. I want to know why do you use text to create turn detection model, not audio likes pipecat? I think that there is a special thing which you did it. Would you give me clarify information.

Thanks in advance!

[KoljaB]

Because that is really hard to do, especially when you need extremely low inference latency for turn detection to be able run the real-time transcription in parallel. You need a ton of correctly labeled audio data upfront, in best case you have thousands of examples from a dialogue between two people. The voice from the one that makes the requests or asks questions has to to vary and cover a broad bandwidth of different voices. Otherwise the model won't generalize and adapt to multiple input voices. In an ideal world you'd have one single model for vad, real-time transcription, turn-detection and if possible even speaker-diarization. But even the simple "only do turn-detection" variant requires an enormous amount of not-that-easy-available data.

[phamkhactu]

@KoljaB thank you very much, I understood your ideas.