"Stream not supported for function calling agent"

[wuchengyuan88]

Just a quick question, will streaming output be eventually supported for function-calling agents in later versions of LlamaIndex?

Or is it permanently unsupported?

Currently, the error message when I try agent.stream_chat/ agent.astream_chat is "Stream not supported for function calling agent".

Thank you.

[dosubot[bot]]

@wuchengyuan88 Hello there! I'm here to assist you with any bugs, questions, or becoming a contributor. Let me know how I can help!

Streaming output is currently not supported for function-calling agents in LlamaIndex, as indicated by the NotImplementedError in the stream_step and astream_step methods. However, the presence of classes and methods related to streaming responses, such as StreamingAgentChatResponse and astream_chat, suggests that there might be plans to support streaming output in the future ^[1][2].

To continue talking to Dosu, mention @dosu.

[mattf]

fyi #15079

[mattf]

fyi #15653

[jacksonwen001]

any updates?

[logan-markewich]

@jacksonwen001 @mattf @wuchengyuan88 it probably won't be implemented, due to a LOT of complexities around exposing a generator to the user. To do this, you need to detect when a tool is not being called, which is different for nearly every LLM

The updated way to do this would be to use the new AgentWorkflow for single and multi agents, which has a much more natural (and advanced) streaming API

https://docs.llamaindex.ai/en/stable/examples/agent/agent_workflow_basic/
https://docs.llamaindex.ai/en/stable/understanding/agent/multi_agents/

[mattf]

@logan-markewich +1, partial tool call responses are of questionable value