Repository navigation
Evaluation judge and user simulator default to gemini-2.5-flash, which is no longer available to new users #7649
Description
The evaluation code defaults to gemini-2.5-flash in three places:
src/google/adk/evaluation/eval_metrics.py(JudgeModelOptions.judge_model)src/google/adk/evaluation/simulation/llm_backed_user_simulator.py(LlmBackedUserSimulatorConfig.model)src/google/adk/evaluation/simulation/_llm_audio_user_simulator.py(text generationmodel)
A user who runs an eval or a simulated conversation without setting a model gets that id, while LlmAgent.DEFAULT_MODEL is already gemini-3.5-flash.
On the Gemini API, the deprecations page says the 2.5 models are "not deprecated and will continue to be served until further notice", but that access to them is limited "to users who have actively used them in the past", and it tells new projects to use a newer model. So a new key can fail on the default. On Vertex AI the 2.5 Flash, Pro and Flash-Lite models are listed to retire on 2026-10-20 (taken from the Vertex model versions page, not re-read when writing this).
Two things I could not settle from the docs:
- The simulator configs also set
ThinkingConfig(include_thoughts=True, thinking_budget=10240). I could not confirm whether Gemini 3.x models acceptthinking_budgetor onlythinking_level, so I did not want to swap the id blindly. - Which id to use.
gemini-3.5-flashwould matchLlmAgent.DEFAULT_MODEL, but the Gemini API deprecations page says requests for it are automatically routed togemini-3.6-flash;gemini-3.8-flashis described as the recommended model for new projects.
Would you prefer that the three defaults follow LlmAgent.DEFAULT_MODEL, or a separate id? And should the thinking config change with it? I can send a small PR once the intended default is clear.
This issue text was drafted with Claude Code.
Activity
inamdarmihir commentedon Oct 26, 2026
I'd like to take this one. I'd switch the three defaults to a single shared constant that follows
LlmAgent.DEFAULT_MODEL(currentlygemini-3.5-flash), so the eval code can't drift from the agent default again. For the simulators'ThinkingConfig(thinking_budget=10240), I'll check what the currentgoogle-genaitypes and the Gemini 3.x docs accept before changing it. If you'd rather pin a different model id, let me know and I'll use that.llalitkumarrr commentedon Oct 26, 2026
Hello @pcbeingused333,
Could you please take a look at #7733 and share your feedback on whether this resolves your issue?
Hello @inamdarmihir,
It looks like you have not signed the CLA. Could you please sign it? Otherwise we will not be able to process your PR further.
pcbeingused333 commentedon Oct 26, 2026
Thanks for the ping. From the diff, #7733 replaces all three
gemini-2.5-flashdefaults from this issue (JudgeModelOptions.judge_model,LlmBackedUserSimulatorConfig.model,LlmAudioUserSimulatorConfig.model) withLlmAgent.DEFAULT_MODEL, and updates the eval config guide table. That covers the part of the issue about a new project getting a model it cannot call, and tying the defaults to the agent default avoids the same drift later. The tests assert the invariant instead of a model name, which is a good choice.Two things the diff alone does not confirm: that importing
LlmAgentfromevaluationcauses no circular import (the PR description says it was checked), and thatThinkingConfig(thinking_budget=10240)is accepted by the model the default now points to (the PR says the Gemini 3 docs keepthinking_budgetfor backward compatibility). If a maintainer knows of a reason to preferthinking_levelfor these simulators, that would be worth settling here. Otherwise, as far as the diff shows, this resolves the issue.Disclosure: this comment was drafted with Claude Code.
inamdarmihir commentedon Oct 26, 2026
Hi @llalitkumarrr, I've signed the Google Individual CLA (as mihirsinamdar@gmail.com), so the CLA check should pass now.
I noticed #7733 already switches the three defaults to
LlmAgent.DEFAULT_MODEL. I'd like to avoid a duplicate PR. Is there a part of this you'd like me to take on, for example verifying thatThinkingConfig(thinking_budget=10240)works with the Gemini 3.x default in the simulators? Otherwise I'll leave this to #7733.