Context Limits for Web Content Synthesis and Accountability

[FarzadKhosravi]

We're trying to build agents that can find and assess quality content on the web based on a user's query. Then, another set of agents creates a synthesis based on this, comparing and contrasting the sources. Lastly, another set of agents that we call the 'accountability' agents (think of it as an editorial board) determines if the final synthesis is 'good'.
Imagine, we're talking about dozens, not hundreds, of articles, videos, podcasts. Each with thousands of words.

What we cannot figure out is how the Autogen framework augments, improves, expands on, or whatever, the existing context limits of the LLMs we use. Especially in the group chat frameworks. Whether it's the 128K GPT 4 Turbo or the 200k Claude 2.1.

We're having a hard time figuring this out. I.e., does the autogen groupchat and other content have its own context window that acts separately from the specific agent that assesses a piece of content for quality, or the agent that compares and contrasts? Or, will our assessment agent's assessment be forgotten by the accountability agent, because the content that has been taken in has gone past the token limit?

[rickyloynd-microsoft]

This could be approached as a RAG application, with additional filtering of sources by a special agent looking at each source in turn and rating its relevance, for instance. There is no magic in group chat to expand or get around the context window size limitation of a given LLM.

[tsindot]

Here's a basic example of how perhaps MemGPT could be used in conjunction with Autogen agents to achieve perpetual conversations and extended context:

Please note that this is a high-level overview, and the actual implementation would require careful design, training, and integration of the various components, including the LLMs, MemGPT, and the Autogen framework.

Autogen Agent for Information Retrieval: This agent uses an LLM, possibly fine-tuned for information retrieval, to retrieve relevant content from the web or an existing database based on the user's query. It generates a list of candidate articles, videos, or podcasts, which are then passed to the next step.

MemGPT Agent for Context Management: This agent uses MemGPT to manage the context of the conversation. It receives the list of candidate sources from the retrieval agent and uses MemGPT's caching and vector database mechanisms to store and retrieve relevant information. It ensures that critical information is not lost beyond the LLM's context window.

Quality Assessment Agent: This agent, possibly using another LLM or a human reviewer, assesses the quality of the content retrieved by the retrieval agent and provides a score or ranking. The assessment takes into account factors such as credibility, relevance, and logical coherence.

Synthesis Generation Agent: This agent, using yet another LLM or a specialized synthesis model, generates a synthesis of the retrieved content based on the user's query and the quality assessments received from the previous step. It compares and contrasts the various sources and produces a coherent summary, leveraging the extended context provided by the MemGPT agent.

Accountability Agent: This agent, acting as an editorial board, uses a human reviewer or another LLM (potentially with specialized training) to determine if the final synthesis is of good quality. It has access to the user's query, retrieved content, quality assessments, and the generated synthesis. The agent ensures that the synthesis aligns with the assessments and makes the final decision on whether to present the synthesis to the user.

In the example, the Autogen framework enables the different agents to work together and leverage MemGPT's capabilities to manage context and create perpetual conversations. The MemGPT agent acts as a key component in extending the context window and ensuring that important information is not lost over time.

[FarzadKhosravi]

Wow, this is super helpful. Thank you so much for your insights
!

[rickyloynd-microsoft]

It would be great to see such MemGPT agents working together in autogen!