Skip to content

evals: sharing CLOUD_ML_REGION between claude and gemini breaks the default gemini model #42

Description

@harche

Problem

evals/run.sh passes CLOUD_ML_REGION into the gemini container as LIGHTSPEED_PROVIDER_REGION (introduced in #40). That variable is shared with the claude via Vertex setup, where its usual value is us-east5. The sandbox app maps it into GOOGLE_CLOUD_LOCATION (config.py#L102), so a region intended for claude also pins the gemini SDK.

The default GEMINI_MODEL in run.sh is gemini-3.1-pro-preview, which is currently only served from the global Vertex location. Result: anyone with the claude Vertex environment exported who runs both providers from the same shell gets a working claude container and a broken gemini container.

Evidence

Tested with the lightspeed-agentic-sandbox image and Vertex ADC:

  1. Gemini container with CLOUD_ML_REGION unset: resolves SDK=gemini, query round trips, {"success":true,"summary":"pong"}. The SDK routes to a location where the model is available.
  2. Gemini container with CLOUD_ML_REGION=us-east5: query fails with 404 NOT_FOUND: Publisher model projects/.../locations/us-east5/publishers/google/models/gemini-3.1-pro-preview was not found.
  3. Direct Vertex generateContent probes: gemini-3.1-pro-preview returns 404 in us-east5 and us-central1, 200 only in global. gemini-2.5-pro and gemini-2.5-flash return 200 in us-east5 today, but model to region availability is Google's call and shifts over time, so the fix should not depend on any particular model and region pairing.

Suggested fix

Give gemini its own region variable instead of sharing CLOUD_ML_REGION with claude, for example LIGHTSPEED_PROVIDER_REGION="${GEMINI_REGION:-}" for the gemini case in evals/run.sh. Left empty, the SDK picks a location where the configured model exists (verified working). Anyone needing a specific region can set GEMINI_REGION explicitly.

Context

Follow up from the review discussion on #40: #40 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions