Skip to content

Per-request Authorization and X-Server-Timeout leak into the client's default headers (stale token in Interactions and Live) #3158

Description

@AtulJoshi1206

Description

When a request is sent without per-request http_options, BaseApiClient._build_request reuses the client's own self._http_options.headers dict for the request. The request path then writes per-request values into it:

  • X-Server-Timeout, via populate_server_timeout_header (_api_client.py ~1545);
  • Authorization and x-goog-user-project (~1564–1566 and ~1641–1645).

Those values become client defaults, and other code reads them as such:

  • Interactions bridge: it snapshots the client's headers (_gaos/google_genai.py ~99–103), and _apply_default_headers copies the stale Authorization onto every request. _apply_auth then skips the fresh token because a header is already set.
  • Live: live.connect only sets Authorization if it's absent (live.py ~1141).
  • Server timeout: populate_server_timeout_header only sets X-Server-Timeout if it's absent.

Impact

  • Vertex AI with ADC or credentials: if any REST call (e.g. models.generate_content) runs before client.interactions (or the other Interactions-layer resources) is first used, those requests send the first bearer token for the life of the process. Once that token expires, after about an hour, they start failing with 401. Long-running servers hit this.
  • Live: it connects with the token from the last REST call, even after a refresh.
  • Timeouts: with a client-level timeout, a later per-request timeout still sends the earlier X-Server-Timeout, and the server enforces it. So the same call behaves differently depending on what ran before.

Repro (offline, mocked transport, fake credentials)

import httpx
import google.oauth2.credentials
from google import genai
from google.genai import types

GEN = {'candidates': [{'content': {'role': 'model', 'parts': [{'text': 'ok'}]}, 'finishReason': 'STOP'}]}
INTERACTION = {'id': 'abc', 'status': 'completed', 'outputs': [], 'object': 'interaction',
               'created': '2026-01-01T00:00:00Z', 'updated': '2026-01-01T00:00:00Z'}
sent = []


def handler(request: httpx.Request) -> httpx.Response:
  kind = 'interactions' if '/interactions/' in request.url.path else 'models'
  sent.append((kind, request.headers.get('authorization')))
  return httpx.Response(200, json=INTERACTION if kind == 'interactions' else GEN)


creds = google.oauth2.credentials.Credentials(token='token-1')
client = genai.Client(
    vertexai=True, project='p', location='us-central1', credentials=creds,
    http_options=types.HttpOptions(httpx_client=httpx.Client(transport=httpx.MockTransport(handler))),
)

client.models.generate_content(model='gemini-2.5-flash', contents='hi')
creds.token = 'token-2'  # what google-auth does when it refreshes the token
client.interactions.get(id='abc')
client.models.generate_content(model='gemini-2.5-flash', contents='hi')
creds.token = 'token-3'
client.interactions.get(id='abc')

for kind, auth in sent:
  print(f'{kind:13s} {auth}')
print('client default headers:', {k: v for k, v in client._api_client._http_options.headers.items() if k.lower() in ('authorization', 'x-server-timeout')})

Output on main (364889a, google-genai 2.31.0):

models        Bearer token-1
interactions  Bearer token-1
models        Bearer token-2
interactions  Bearer token-1      <- credentials hold token-3
client default headers: {'Authorization': 'Bearer token-2'}

If interactions is used before any models call, it gets the fresh token each time, so this depends on call order.

Proposed fix

Copy the headers at the one place that builds per-request headers, so per-request values stay on the HttpRequest:

@@ class BaseApiClient: def _build_request(
     if patched_http_options.headers is None:
       raise ValueError('Request headers must be set.')
-    populate_server_timeout_header(
-        patched_http_options.headers, timeout_in_seconds
-    )
+    # Without per-request options, patched_http_options is the client's own
+    # options. Copy the headers so that this request's X-Server-Timeout, and
+    # the Authorization header set when it is sent, do not become client
+    # defaults that later requests, Interactions and Live reuse.
+    headers = dict(patched_http_options.headers)
+    populate_server_timeout_header(headers, timeout_in_seconds)
     return HttpRequest(
         method=http_method,
         url=url,
-        headers=patched_http_options.headers,
+        headers=headers,
         data=request_dict,
         timeout=timeout_in_seconds,
     )

With this, Interactions and Live always send the current token, X-Server-Timeout matches each request's timeout, and the client's default headers stay as configured. It covers sync and async and every transport.

I'd rather not patch the readers (filter Authorization in the bridge, always overwrite X-Server-Timeout). That would be three separate patches, and it would break the existing rule that a user-set X-Server-Timeout wins (test_server_timeout_not_overwritten).

One behaviour change worth noting: on Vertex, Live currently only gets x-goog-user-project incidentally, after some REST call has written it into the shared headers. With this fix it won't, so Live should set it itself from credentials.quota_project_id.

I have regression tests that fail on main:

  • a unit test on _build_request;
  • sync and async models → Interactions;
  • models → Live.

Happy to open a PR.

Activity

  1. Priyanshujindal commented on Oct 17, 2026

    @Priyanshujindal

    Can you assign me to this issue

  2. Priyanshujindal commented on Oct 17, 2026

    @Priyanshujindal

    #3159

    this is my pr

  3. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Oct 19, 2026
  4. Venkaiahbabuneelam commented on Oct 19, 2026

    @Venkaiahbabuneelam

    Hi @AtulJoshi1206, thank you for the detailed bug report, reproduction script, and root cause analysis!
    We have reproduced this issue (Authorization and X-Server-Timeout leaking into client._api_client._http_options.headers, causing stale tokens in client.interactions and live.connect) and escalated it to our engineering team.
    Thanks

  5. added 2 commits that reference this issue on Oct 21, 2026
    6f36e1b
    207a1dd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions