Repository navigation
fix(auth): bound the OAuth M2M token request - #557
Merged
Merged
Conversation
vuanhphung
marked this pull request as ready for review
November 13, 2026 16:25
This was referenced Nov 13, 2026
The M2M authenticator built its token source on context.Background(), so oauth2 sent token requests through http.DefaultClient, which has no timeout. A token endpoint that accepted the connection but never answered blocked Authenticate, and every request queued behind its mutex, forever. Give the token exchange a copy of the context's oauth2 HTTP client (or http.DefaultClient) with its Timeout capped at 60s, the SQL kernel's default HTTP timeout. The token source keeps that context, so refreshes are bounded too. A client supplied on the context keeps its transport and any shorter timeout. The exported GetTokenSource is unchanged. Addresses #475 Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
vuanhphung
force-pushed
the
fix/issue-475-m2m-token-timeout
branch
from
November 20, 2026 00:57
fa87771 to
8420441
Compare
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OAuth M2M authenticator builds its token source on
context.Background(), sooauth2sends token requests throughhttp.DefaultClient, which has no timeout. If a token endpoint accepts the connection but never answers,Authenticateblocks forever. So does every request queued behind its mutex: Thrift calls, feature flags and telemetry all authenticate through it. The caller's context can't help, because the token source never sees it.This PR gives the token exchange a copy of the context's
oauth2.HTTPClient(http.DefaultClientwhen unset), withTimeoutcapped at 60s. That's the SQL kernel's default total HTTP timeout; Go's default transport already caps connect at 30s, as the kernel does. The token source keeps that context, so hourly refreshes are bounded too. Until a fetch succeeds, oauth2's AuthStyle auto-detect retries a failed request once, so the worst case for such a fetch is about 2 × 60s. After the first success the detected style is cached, so a refresh makes one request. The exportedGetTokenSourceis unchanged;Authenticateno longer calls it. The mutex scope is unchanged too:oauth2.ReuseTokenSourceserializes refreshes under its own lock, so narrowingc.mxwouldn't unblock anyone. The timeout is what bounds the stall.This is a per-request cap, not a 60s RPC deadline. Retryable Thrift RPCs such as
OpenSessionrepeat failed token fetches underWithRetries: with the defaults (five attempts, 15s total backoff), a continuously hung endpoint can take about 615s before failure, excluding discovery and waiting behind other callers. Caller cancellation still does not interruptAuthenticate. AuthStyle auto-detection remains unchanged for compatibility with either credential placement. The HTTP client is copied when the token source is created: longer timeouts are capped, and subsequent replacement ofhttp.DefaultClientor changes to its Jar/CheckRedirect are not picked up by refreshes. The exportedGetTokenSourceretains its existing unbounded-default behavior; callers using it directly must configure their own HTTP timeout.How this composes with #562 (successor to #498). The cap is applied to whatever client is already on the context. A scratch merge with #562 @2877915 installs its connector-local client first and then applies the 60s cap on every uncached grant/refresh, preserving shorter timeouts, transport/User-Agent and the shared token cache. Both PRs'
auth/...andinternal/clienttests pass, including race x3 and composition cases with 0, 900s and 50ms clients. Neither PR changes the other's branch. Main's new ExecuteStatement dial-only retry remains safe: a token read timeout is not retried as a SQL dial error; a token dial failure occurs before any SQL send (production Transport/RetryPolicy tests, race x3).Verification (rebased onto d034123, November 20). The three
httptestregressions still fail on main with only discovery/test hooks retained and the timeout fix reverted: each hits its 10s guard. At the 200ms test bound the rebased branch returns timeout errors in 0.40s (first fetch), 0.80s (two concurrent callers), and 0.20s (refresh).make testpasses (1694 tests, 9 credential-gated skips),make lintreports 0 issues, and auth/client race tests pass x3. Pure-Go and kernel-tagged builds pass with Go 1.27.3.The full kernel unit suite has one pre-existing failing test,
TestKernelCloudFetchDisabledPreservesUnsupportedServerError(four subcases), identical on main and branch: the pinned v1.4.0 path hits the mock server'sHTTP 400: missing disabled download capabilitycheck. All other kernel unit tests pass.gofmt -s -l .lists the same three generatedinternal/cli_servicefiles on main and branch; changed Go files are formatted. No unrelated/generated files were changed.Live Thrift e2e 4/4 and kernel e2e 4/4 pass without skips. The kernel M2M path still consumes raw credentials and is unaffected. A successful live M2M connect was not run because no service-principal secret is available. Real token-endpoint smoke with invalid client credentials returns the same
*oauth2.RetrieveError, HTTP 401invalid_client, on both sides: three alternating fresh-process runs each, median Authenticate 309.80ms (main) / 323.58ms (branch), Ping 238.41ms / 215.64ms. The warehouse is shared.Performance: six alternating fresh-process microbenchmark runs per side, 500ms samples, rebuilding in alternating order. Cached Authenticate: median 141.45ns → 142.10ns, unchanged 40 B / 2 allocations (benchstat p=0.102). First HTTP token exchange: 76.85µs → 78.66µs (p=0.699), 11.49 → 13.41 KiB and 136 → 160 allocations from the timeout/context/timer machinery. No statistically significant timing change; only uncached exchanges pay the additional work. Benchstat source was pinned to be2c69f through the approved proxy because its
@latestlookup was unavailable.The other parts of #475 are not changed here.
QueryContextcancellation (Fix context loss in polling and connection close operations #295/[ES-1934053] Detach result streaming from QueryContext cancellation #371/[ES-1934053] Detach result streaming from QueryContext cancellation #373, guarded byTestE2EArrowBatchesSurviveQueryContextCancellation).rows.Close()cancels downloads.MinTimeToExpiryoption are new features.Parts 2 and 3 are deferred for an API decision.
Cross-driver state
src/client/http.rs:302-303,519-522@8877579). Sessions userequest_timeout(120s default; Go forwardsClientTimeout); 30s Auth retry budget does not bound an active attemptsocketTimeoutdefault 900s (DatabricksJdbcUrlParams.java:165,ClientConfigurator.java:60@97dac8d); SDK applies connect/socket/pool timeouts, wraps failure inDatabricksException(@94c2446)credentials_providerstill callsrequests.postwithout a timeout (oauth.py:209@b82cb0e; driver @087b2bf); SEA uses the kernelRPError; driver overrides only agent (OAuthManager.ts:70-72,103@1c499e2)Adversarial review: round 1 converged with no high/medium findings; clarified retry-inclusive timing and client-snapshot behavior, retaining existing auth-style and exported-helper semantics.
Addresses #475
This pull request and its description were written by Isaac.