Max output reach

[TheGuyD]

I have an agent that output structured output, the thing is when the output is above 65k (the maximum gemini output) the agent run stop and return non-valid json.

I wander how can I tackle this problem so that I'll be able to eventually get the complete output (not just the valid json but the complete output).

Thanks.

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: You can use streaming to handle large outputs from the Gemini API that exceed the maximum token limit. Instead of receiving the entire response at once, you can process it in smaller chunks as it's generated.

---

When you encounter issues with the output size limit of the Gemini API, the recommended approach is to stream the response. This allows you to receive and process the output in smaller parts, avoiding the problem of hitting the maximum token limit for a single response.

Here is a Python code example of how to stream the output from the Gemini model:

import google.generativeai as genai

# ... (your existing code for model initialization)

response = model.generate_content("Write a very long story", stream=True)

for chunk in response:
  print(chunk.text)

By setting stream=True in the generate_content method, you can iterate over the response chunks as they become available. This is a more efficient way to handle large outputs and should resolve the issue of getting an incomplete or invalid JSON response. [7]

For more details, you can refer to the official documentation on text generation with the Gemini API. [5]

---

Citations:
[5] Text generation | Gemini API | Google AI for Developers (https://ai.google.dev/gemini-api/docs/text-generation)
[7] Migrate to the Google GenAI SDK | Gemini API | Google AI for ... (https://ai.google.dev/gemini-api/docs/migrate)

[xuanyang15]

I doubt using streaming is solution for this issue. @TheGuyD Is it possible to break down the question, generate sub structured output, and synthesize them at a later stage?

[dylan-apex]

Similar to what xuanyang recommends, we use Gemini for entity extraction on very large pdfs and we have to 1) split the pdf into 10-20 page chunks and 2) run multiple prompts with different output_schemas defined and then coalesce them together later.