Websocket streaming audio in realtime from client

[revmag]

Please read this first

• Have you read the docs?Agents SDK docs
• Have you searched for related issues? Others may have had similar requests

Describe the feature

What is the feature you're requesting? How would it work? Please provide examples and details if possible.

Currently, the OpenAI Agents SDK provides excellent support for terminal-based streaming voice applications, but lacks an official way to implement real-time audio streaming between web clients and a server.

I'd like to request a WebSocket adapter that enables browser-based clients to:

1. Stream audio to the server in real-time
2. Receive TTS audio responses in real-time
3. Maintain the same low-latency experience as the terminal examples

[rm-openai]

Would this work? https://github.com/openai/openai-voice-agent-sdk-sample

[sid-js]

@rm-openai Is there any example with Voice Activity Detection to detect turns? The above sample implements only push to talk.

[rm-openai]

@dkundel-openai ?

[pmmohanmishra]

@dkundel-openai voice agents are becoming quintessential with every passing day. The agents sdk doesn’t seem to be cut out for it though. Multiple issues related to output delay, language issues, no realtime support

[dkundel-openai]

Hey @sid-js we don't have a good sample app for this right now but the sample Rohan mentioned can be modified to fit your needs. Namely:

1. Inside the server.py you need to:
   a. Create a StreamedAudioInput instance that acts as your audio input at the beginning of the session
   b. Start running your voice pipeline with that StreamedAudioInput instead of the audio input buffer
   c. Change is_new_audio_chunk to append the incoming audio to the StreamedAudioInput.
   d. This should result in the Agents SDK handling VAD using either semantic_vad (default) or server_vad depending on what you configure
2. On the front-end you want to:
   a. Expose the audioChunks from the useAudio hook
   b. Trigger sendAudioMessage to send on every new audio chunk
   c. Remove from sendAudioMessage the sending of the input_audio_buffer.commit event

Hope that helps. We are working on making building end to end voice agent experiences better especially with Realtime API support