Skip to content

fix(auth): bound the OAuth M2M token request - #557

Merged
vuanhphung merged 3 commits into
mainfrom
fix/issue-475-m2m-token-timeout
Nov 20, 2026
Merged

vuanhphung merged 3 commits into
mainfrom
fix/issue-475-m2m-token-timeout

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

The OAuth M2M authenticator builds its token source on context.Background(), so oauth2 sends token requests through http.DefaultClient, which has no timeout. If a token endpoint accepts the connection but never answers, Authenticate blocks forever. So does every request queued behind its mutex: Thrift calls, feature flags and telemetry all authenticate through it. The caller's context can't help, because the token source never sees it.

This PR gives the token exchange a copy of the context's oauth2.HTTPClient (http.DefaultClient when unset), with Timeout capped at 60s. That's the SQL kernel's default total HTTP timeout; Go's default transport already caps connect at 30s, as the kernel does. The token source keeps that context, so hourly refreshes are bounded too. Until a fetch succeeds, oauth2's AuthStyle auto-detect retries a failed request once, so the worst case for such a fetch is about 2 × 60s. After the first success the detected style is cached, so a refresh makes one request. The exported GetTokenSource is unchanged; Authenticate no longer calls it. The mutex scope is unchanged too: oauth2.ReuseTokenSource serializes refreshes under its own lock, so narrowing c.mx wouldn't unblock anyone. The timeout is what bounds the stall.

This is a per-request cap, not a 60s RPC deadline. Retryable Thrift RPCs such as OpenSession repeat failed token fetches under WithRetries: with the defaults (five attempts, 15s total backoff), a continuously hung endpoint can take about 615s before failure, excluding discovery and waiting behind other callers. Caller cancellation still does not interrupt Authenticate. AuthStyle auto-detection remains unchanged for compatibility with either credential placement. The HTTP client is copied when the token source is created: longer timeouts are capped, and subsequent replacement of http.DefaultClient or changes to its Jar/CheckRedirect are not picked up by refreshes. The exported GetTokenSource retains its existing unbounded-default behavior; callers using it directly must configure their own HTTP timeout.

How this composes with #562 (successor to #498). The cap is applied to whatever client is already on the context. A scratch merge with #562 @2877915 installs its connector-local client first and then applies the 60s cap on every uncached grant/refresh, preserving shorter timeouts, transport/User-Agent and the shared token cache. Both PRs' auth/... and internal/client tests pass, including race x3 and composition cases with 0, 900s and 50ms clients. Neither PR changes the other's branch. Main's new ExecuteStatement dial-only retry remains safe: a token read timeout is not retried as a SQL dial error; a token dial failure occurs before any SQL send (production Transport/RetryPolicy tests, race x3).

Verification (rebased onto d034123, November 20). The three httptest regressions still fail on main with only discovery/test hooks retained and the timeout fix reverted: each hits its 10s guard. At the 200ms test bound the rebased branch returns timeout errors in 0.40s (first fetch), 0.80s (two concurrent callers), and 0.20s (refresh). make test passes (1694 tests, 9 credential-gated skips), make lint reports 0 issues, and auth/client race tests pass x3. Pure-Go and kernel-tagged builds pass with Go 1.27.3.

The full kernel unit suite has one pre-existing failing test, TestKernelCloudFetchDisabledPreservesUnsupportedServerError (four subcases), identical on main and branch: the pinned v1.4.0 path hits the mock server's HTTP 400: missing disabled download capability check. All other kernel unit tests pass. gofmt -s -l . lists the same three generated internal/cli_service files on main and branch; changed Go files are formatted. No unrelated/generated files were changed.

Live Thrift e2e 4/4 and kernel e2e 4/4 pass without skips. The kernel M2M path still consumes raw credentials and is unaffected. A successful live M2M connect was not run because no service-principal secret is available. Real token-endpoint smoke with invalid client credentials returns the same *oauth2.RetrieveError, HTTP 401 invalid_client, on both sides: three alternating fresh-process runs each, median Authenticate 309.80ms (main) / 323.58ms (branch), Ping 238.41ms / 215.64ms. The warehouse is shared.

Performance: six alternating fresh-process microbenchmark runs per side, 500ms samples, rebuilding in alternating order. Cached Authenticate: median 141.45ns → 142.10ns, unchanged 40 B / 2 allocations (benchstat p=0.102). First HTTP token exchange: 76.85µs → 78.66µs (p=0.699), 11.49 → 13.41 KiB and 136 → 160 allocations from the timeout/context/timer machinery. No statistically significant timing change; only uncached exchanges pay the additional work. Benchstat source was pinned to be2c69f through the approved proxy because its @latest lookup was unavailable.

The other parts of #475 are not changed here.

  1. CloudFetch deliberately detaches result streaming from QueryContext cancellation (Fix context loss in polling and connection close operations #295/[ES-1934053] Detach result streaming from QueryContext cancellation #371/[ES-1934053] Detach result streaming from QueryContext cancellation #373, guarded by TestE2EArrowBatchesSurviveQueryContextCancellation). rows.Close() cancels downloads.
  2. A CloudFetch download timeout or client option needs new public API. (CloudFetch and M2M token exchange are unbounded and uninterruptible (v1.13.0) (#475) #522, which added one, was closed.)
  3. Re-fetching expired links and a MinTimeToExpiry option are new features.

Parts 2 and 3 are deferred for an API decision.

Cross-driver state

Driver State Notes
Kernel Supported Default connect 30s / total 60s (src/client/http.rs:302-303,519-522 @8877579). Sessions use request_timeout (120s default; Go forwards ClientTimeout); 30s Auth retry budget does not bound an active attempt
JDBC Supported socketTimeout default 900s (DatabricksJdbcUrlParams.java:165, ClientConfigurator.java:60 @97dac8d); SDK applies connect/socket/pool timeouts, wraps failure in DatabricksException (@94c2446)
Python Not supported Thrift SDK credentials_provider still calls requests.post without a timeout (oauth.py:209 @b82cb0e; driver @087b2bf); SEA uses the kernel
Node.js Supported openid-client 5.4.2 default 3.5s timeout / RPError; driver overrides only agent (OAuthManager.ts:70-72,103 @1c499e2)
Go Not supported → Supported (this PR) 60s per token request, following the kernel's default total; shorter supplied timeout retained, default connect bound already 30s

Adversarial review: round 1 converged with no high/medium findings; clarified retry-inclusive timing and client-snapshot behavior, retaining existing auth-style and exported-helper semantics.

Addresses #475

This pull request and its description were written by Isaac.

@vuanhphung
vuanhphung marked this pull request as ready for review November 13, 2026 16:25

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

The M2M authenticator built its token source on context.Background(), so
oauth2 sent token requests through http.DefaultClient, which has no
timeout. A token endpoint that accepted the connection but never answered
blocked Authenticate, and every request queued behind its mutex, forever.

Give the token exchange a copy of the context's oauth2 HTTP client (or
http.DefaultClient) with its Timeout capped at 60s, the SQL kernel's
default HTTP timeout. The token source keeps that context, so refreshes
are bounded too. A client supplied on the context keeps its transport and
any shorter timeout. The exported GetTokenSource is unchanged.

Addresses #475

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung force-pushed the fix/issue-475-m2m-token-timeout branch from fa87771 to 8420441 Compare November 20, 2026 00:57

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@vuanhphung
vuanhphung enabled auto-merge November 20, 2026 04:04
@vuanhphung
vuanhphung added this pull request to the merge queue Nov 20, 2026
@vuanhphung
vuanhphung removed this pull request from the merge queue due to a manual request Nov 20, 2026
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@vuanhphung
vuanhphung enabled auto-merge November 20, 2026 04:09
@vuanhphung
vuanhphung added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit c1f2469 Nov 20, 2026
14 checks passed
@vuanhphung
vuanhphung deleted the fix/issue-475-m2m-token-timeout branch November 20, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant