How can I stream to a frontend using SSE?

[tyler-suard-parker]

Describe the issue

I would like to host Autogen on Flask, end then stream the responses to a frontend using SSE. I am not sure how to do that. SSE can return streaming messages if it uses a generator, but I am not sure how to get the code out from the deep innards of Autogen (client.py, conversable_agent.py) and bring it up to the top so Flask can return that stream. Any help on this would be much appreciated. @ragyabraham

Steps to reproduce

No response

Screenshots and logs

No response

Additional Information

No response

[ekzhu]

Thanks for the issue! Could you expand a little bit on what SSE is?

[tyler-suard-parker]

Yes, I'm sorry about that, SSE stands for server-sent events, which is, from what I understand, how ChatGPT streams information to its UI.

[ekzhu]

Got it. Check if my understanding is correct: on the server side (Python) you would have to create a data stream as part of the response object with mime type "text/event-stream"? So, something similar to:

@app.route("/stream")
def stream():
    def generate():
        while True:
            yield "data: {}\n\n".format("Hello, World!")
    return Response(generate(), mimetype="text/event-stream")

The generate() function is a generator that when called, produces a stream of string and the front end can receive from. While this generator object is managed by the Flask framework itself.

So the only thing we need to produce from the autogen side is a Python generator that runs in its own thread listening to a concurrent pipe and yields data from that pipe.

I think this can be achieved by registering a middleware (#1240) to the agent you want to stream response from, and every time you catch its response in the middleware you then asynchronously writes the response to the concurrent pipe the generator is listening to. This way your Flaks code you can call the agent and directly returns the generator in your Response object. Of course you need to close the concurrent pipe in the middleware once you have done with it.

You are welcome to contribute to the effort there with an implementation of a middleware that takes input a concurrent pipe, writes response message to that pipe, and has a method to close it once finished.

[tyler-suard-parker]

@ekzhu Thank you for your response. Yes, what you said about creating a generator is correct as far as I understand it. However, I do not understand what you mean by registering a middleware, and I don't understand the example code in #1240 either.

[ekzhu]

Yes. This is a work in progress. Will update later in the PR.

[tyler-suard-parker]

@ekzhu Another idea might be to try capturing and streaming the stdout.

[ekzhu]

Some people have had success hacking a solution by rewriting the _print_recieved_message function to have it writing to a socket. Maybe you can try that as a temporary solution?

[tyler-suard-parker]

Ok, thank you, I will try that. I haven't been able to excavate the print statement from Autogen to put it into a generator so I can use server-side events so I will try using sockets.