Skip to content

feat(llm): use Gemini native generateContent/streamGenerateContent endpoints on Vertex#2023

Open
htimur wants to merge 8 commits into
agentgateway:mainfrom
htimur:feat/vertex-native-gemini
Open

feat(llm): use Gemini native generateContent/streamGenerateContent endpoints on Vertex#2023
htimur wants to merge 8 commits into
agentgateway:mainfrom
htimur:feat/vertex-native-gemini

Conversation

@htimur
Copy link
Copy Markdown

@htimur htimur commented Jun 1, 2026

Fixes #1929.

Summary

This change adds a native Gemini api usage to the Vertex provider. Gemini models are now routed directly to :generateContent and :streamGenerateContent, with request and response translation handled between the OpenAI chat-completions format and Gemini's native API.

No changes for Anthropic models on Vertex were implemented, and usage of OpenAI compatible endpoint remains as it was before.

Worth mentioning / notes

  • Added a new CEL variable and access-log field, llm.upstreamFinishReason, which exposes the original Gemini finish reason before it is mapped to an OpenAI finish reason. Its implemented only for the Vertex Gemini models, and I believe this is important for observability/monitoring.
  • Image urls: data: urls are converted to inline data and gs:// urls to fileData with a resolved MIME type. http(s) urls are rejected because Vertex cannot fetch remote images directly.

What changed

  • Routing: Vertex gemini-* models on the completions route now use :generateContent and :streamGenerateContent for streaming. Gemini embedding models (for example, gemini-embedding-001) continue to use the embeddings endpoint, and Anthropic-on-Vertex behavior is unchanged.
  • Request translation: OpenAI chat-completions requests are translated to Gemini's native format. Messages are mapped to contents (including system instructions, tool calls, and tool responses), tools to functionDeclarations, tool_choice to functionCallingConfig, and sampling, structured output, and reasoning settings to generationConfig (including thinkingConfig). cachedContent, labels, and safetySettings are passed through unchanged.
  • Response translation: Gemini responses and streaming events are translated back into the OpenAI chat-completions format, including finish reasons, tool calls, reasoning content, and token usage.

What was tested

Manual tests and evals were executed against Vertex Gemini models on a locally built instance of the gateway alongside the test suite and google ADK based tests, and no behaviour changes were detected.

Use of AI assistance

The native Gemini wire types and parts of the test suite were developed with the help of an LLM. I have reviewed the generated code, verified my understanding of its behaviour and taken ownership of the implementation. I reviewed the tests for correctness and coverage, and validated the change to confirm that it behaves as expected.

@htimur htimur requested a review from a team as a code owner June 1, 2026 14:42
htimur added 7 commits June 1, 2026 16:48
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
…ructs

Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
…loop roles

Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
@htimur htimur force-pushed the feat/vertex-native-gemini branch from cbf76a7 to 55e4337 Compare June 1, 2026 14:48
@htimur htimur marked this pull request as draft June 1, 2026 15:20
Signed-off-by: Timur Khamrakulov <timur.khamrakulov@gmail.com>
@htimur htimur marked this pull request as ready for review June 2, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vertex: support native generateContent endpoint

1 participant