Repository navigation
Addition of @assistant.on("function_calls_started") #399
Description
Would it be possible to add something like:
@assistant.on("function_calls_started")
We currently have @assistant.on("function_calls_finished")... But some of the functions which I'm calling take a couple of seconds to respond with data.
It would be great to be able to tap into a "starting" state, where I could then:
await assistant.say("Hmmm, let me just look that up for you - Give me 2 seconds.")
Any advice or input would be greatly appreciated
Activity
theomonnom commentedon Jul 26, 2024
Hey, can function_calls_collected satisfy your needs?
willsmanley-archive commentedon Jul 27, 2024
you could just instruct the llm to always return a spoken response along with the function call. this works the majority of the time for me but I've found there are also times when it ignores this instruction despite strong prompting.
or you could include a chat completion request in the body of the function call itself? particularly if some of your function calls require this "hold on" dialogue and some don't.
ChrisFeldmeier commentedon Sep 8, 2024
I don't know if this helps you, but it helped me! This is the way I receive events from the agent server and send it via publish_data to the client app side.
See the full code in attached .txt file here:
agent.txt
assistant = VoiceAssistant( vad=silero.VAD(), stt=deepgram.STT(language="de-DE"), llm=openai.LLM(model="gpt-4o"), tts=elevenlabs.TTS(voice=VOICE1, language="de"), chat_ctx=initial_ctx, allow_interruptions=True, interrupt_volume=0.20, interrupt_speech_duration=0.65, interrupt_min_words=1, base_volume=1.0, transcription_speed=1.83, ) # Start the assistant assistant.start(ctx.room) # Event handlers assistant.on('user_started_speaking', lambda: user_started_speaking(assistant, ctx.room.local_participant)) assistant.on('user_stopped_speaking', lambda: user_stopped_speaking(assistant, ctx.room.local_participant)) assistant.on('agent_started_speaking', lambda: agent_started_speaking(assistant, ctx.room.local_participant)) assistant.on('agent_stopped_speaking', lambda: agent_stopped_speaking(assistant, ctx.room.local_participant)) assistant.on('agent_speech_interrupted', lambda: agent_stopped_speaking(assistant, ctx.room.local_participant)) # Register data received handler ctx.room.on("data_received", lambda dp: asyncio.create_task(handle_data_received(dp, assistant))) # Register the stop_speaking event handler assistant.on('stop_speaking', lambda: stop_speaking(assistant))zamia commentedon Oct 15, 2024
I met the same scenario, have you solve it? any suggestions would be great. @birksy89
birksy89 commentedon Oct 16, 2024
@zamia I'm currently waiting for the Node version of the API to be stabilised a bit more. I don't know Python, and I think a lot of the friction I was getting may be partly down to that.
There's the new realtime API which is a different paradigm, and hopefully can solve this
zamia commentedon Oct 16, 2024
Thanks @birksy89
I solve it by using LLM prompt and let LLM attach a text content along with function call. It is supported in the newest version of livekit-agents.
I hope it helps you too.
birksy89 commentedon Oct 17, 2024
@zamia can you provide a snippet for example?
Does the LLM return the text before the function call is finished?
zamia commentedon Oct 17, 2024
the response text is along side with the function call, they are in the same response. like this( this log is from claude-3.5-sonnet)
the system prompt is like this:
I also found gpt-4o-mini does not work properly, but gpt-4o and claude-3.5-sonnet works fine.
birksy89 commentedon Oct 18, 2024
@zamia thanks for the reply.
Yeah, this is what I ended up doing originally - But the response came always just before the function call.
So there was a gap, (while the function was processed) and then the "let me check" + function call response came together shortly after.
It was probably the model which was being used. I was trying these things very early in the release.
I personally still think there's a plan for the originally proposed:
@assistant.on("function_calls_started")Hopefully one day someone comes along with a solid solution - Or the new realtime API handles this differently (I'm yet to dig into that too much yet)
tinalenguyen commentedon Jul 13, 2026
added in #6100