Merge pull request #560 from koic/let_a_provider_customize_the_oauth_http_client

Let a provider customize the HTTP client the OAuth flow uses

Author: koic

docs/_client/authorization.md

@@ -97,6 +97,8 @@ Optional keyword arguments:
 - `scope`: Space-separated scopes to request when the server's `WWW-Authenticate` does not specify one.
 - `authorization_request_validator`: Callable invoked with an `MCP::Client::OAuth::AuthorizationRequest` before any authorization request is built.
   Returning a falsy value abandons the flow with `Flow::AuthorizationRefusedError`. See [Reviewing the authorization request](#reviewing-the-authorization-request).
+- `http_client_customizer`: Callable invoked with the Faraday connection the SDK builds for the OAuth flow's own requests, after its defaults and before its origin guard.
+  See [Customizing the OAuth HTTP Client](#customizing-the-oauth-http-client).
 - `storage`: Object responding to `tokens`, `save_tokens(t)`, `client_information`, `save_client_information(info)`. Defaults to `MCP::Client::OAuth::InMemoryStorage`,
   which keeps credentials in process memory only. Persisted `client_information` is stamped with an `"issuer"` member binding it to the authorization server that
   issued it (SEP-2352): when the server's authorization server changes, the SDK discards the stale registration and its tokens and re-registers automatically
@@ -219,7 +221,7 @@ Keyword arguments:
 - `private_key`, `signing_algorithm`: Required with `private_key_jwt` - the key (a PEM string
   or `OpenSSL::PKey::PKey`, never written to `storage`) signs the client assertion with `"ES256"`
   or `"RS256"`; `client_secret` must not be set, because the private key is the credential.
-- `scope`, `storage`, `authorization_request_validator`, `token_request_params`: Optional, same meaning as on `Provider`.
+- `scope`, `storage`, `authorization_request_validator`, `token_request_params`, `http_client_customizer`: Optional, same meaning as on `Provider`.
   Use `token_request_params` for a parameter the authorization server requires on the `client_credentials` grant, such as Auth0's `audience`.
 
 ### Cross-App Access (JWT Bearer) Grant
@@ -258,7 +260,39 @@ Keyword arguments:
 - `assertion_provider`: Required. Callable invoked as `call(audience:, resource:)` and returning the ID-JAG assertion.
   `audience` is the MCP authorization server's validated issuer identifier; `resource` is the canonical MCP server URL (RFC 8707).
   Passing both through to `IDJAGTokenExchange.request` covers the common case.
-- `scope`, `storage`, `authorization_request_validator`, `token_request_params`: Optional, same meaning as on `Provider`.
+- `scope`, `storage`, `authorization_request_validator`, `token_request_params`, `http_client_customizer`: Optional, same meaning as on `Provider`.
+
+### Customizing the OAuth HTTP Client
+
+The requests the OAuth flow makes (Protected Resource Metadata discovery on the MCP server's origin, authorization server metadata discovery,
+dynamic client registration, and every token request the flow sends, whether the first exchange, a refresh, or a step-up) go over a Faraday connection of their own,
+not over the transport's connection: the transport's is bound to the MCP server URL and carries the `headers:` and the customizer block meant for that server.
+To add middleware to the OAuth flow's connection, or to swap its adapter, pass `http_client_customizer:` to the provider:
+
+```ruby
+provider = MCP::Client::OAuth::ClientCredentialsProvider.new(
+  client_id: "my-service",
+  client_secret: ENV.fetch("MCP_CLIENT_SECRET"),
+  http_client_customizer: ->(faraday) { faraday.use MyApp::Middleware::HttpRecorder },
+)
+```
+
+The callable receives the `Faraday::Connection` after the SDK has applied its defaults and registered the middleware that records the requested URL,
+and before the SDK registers its origin guard, the same position the transport's customizer block has on the MCP server connection.
+It may be invoked more than once, and from more than one thread at a time, so keep it free of side effects and safe to run concurrently;
+today it runs once per authorization attempt, but that is not a promise.
+A few constraints follow from the checks described below:
+
+- Do not add redirect-following middleware. Every destination check runs against the URL as written, so a request that middleware added by the customizer would send
+  to a different origin after the SDK has recorded the requested URL, whether by following a `3xx` or by rewriting the URL, is refused with `Flow::DestinationMismatchError`
+  before it reaches the adapter, as is a request that reaches the guard without that record. Middleware inserted ahead of the record with `builder.insert(0, ...)`
+  that rewrites the URL first or rebuilds the environment is outside the guard, as is following done inside an adapter, so leave both off.
+- Leave `Accept-Encoding` unset. The response cap below is measured on decoded bytes, and claiming the header turns Net::HTTP's decoding off.
+- Do not add Faraday's `raise_error` middleware. The flow reads statuses itself, both to tell an absent metadata document from a failed request
+  and to turn a token endpoint error into `Flow::InvalidGrantError`.
+- With an adapter that does not stream through `on_data`, the response cap is applied once the body has been buffered rather than as it arrives.
+- A middleware that records requests sees the client credentials on token requests (`Authorization: Basic`, `client_secret`, `client_assertion`), refresh tokens,
+  and the access tokens in token responses; redact them before they reach a log.
 
 ### Communication Security
 
@@ -290,8 +324,11 @@ The range check compares IP literals and does not resolve hostnames, so it canno
 such as `https://vault.corp.internal/`. Resolving names here would not close that gap either, because the address the SDK looked up need not be the one
 the HTTP client connects to a moment later. The same-origin rule is what protects the `resource_metadata` URL, which is the only one of these a server supplies directly.
 
-If you replace the OAuth HTTP client through `MCP::Client::OAuth::Flow.new(http_client_factory:)`, do not add redirect-following middleware. Every check above runs against
-the URL as written, so a connection that follows a `3xx` on its own would reach hosts these rules just refused.
+On the connection the SDK builds, a middleware that would send a request to a different origin, by following a `3xx` or by rewriting the URL, is refused before the request
+goes out (see [Customizing the OAuth HTTP Client](#customizing-the-oauth-http-client)). A connection supplied through `MCP::Client::OAuth::Flow.new(http_client_factory:)`
+replaces that one, the provider's `http_client_customizer` and the guard included, so do not add redirect-following middleware to it: every check above runs against
+the URL as written, and a connection that follows a `3xx` on its own would reach hosts these rules just refused.
+A factory that wants to keep them can return `MCP::Client::OAuth::Flow.build_http_client(customizer)`, the connection the SDK builds for itself.
 
 The SDK also bounds what those endpoints may return. A discovery, dynamic client registration, token, or token exchange response is refused once it passes 4 MiB,
 measured as the body arrives rather than after it has been buffered, so a compressed body that expands past the limit is refused partway through the expansion.

docs/_client/transports.md

@@ -248,6 +248,9 @@ http_transport = MCP::Client::HTTP.new(url: "https://api.example.com/mcp") do |f
 end
 ```
 
+The block customizes only the connection to the MCP server. The connection the OAuth flow uses for its own requests is customized through
+the provider's `http_client_customizer:` keyword instead; see [Customizing the OAuth HTTP Client](/client/authorization/#customizing-the-oauth-http-client).
+
 {: .note }
 > Answers to server-to-client requests (a pong, an elicitation result) are POSTed from inside
 > the SSE streaming callback of another response, re-entering the connection on the same thread.

lib/mcp/client/oauth/bounded_body.rb

@@ -26,7 +26,7 @@ def initialize(max_bytes: MAX_RESPONSE_BYTES)
         # Faraday `on_data` streaming callback. The chunks arrive decompressed: the default `Net::HTTP` adapter negotiates
         # `Accept-Encoding` itself and reads the body through `Net::HTTPResponse#inflater`, so a small compressed body
         # that expands past the cap is refused partway through the expansion rather than after it. That holds only while
-        # the connection leaves `Accept-Encoding` to the adapter; see `Flow#default_http_client`.
+        # the connection leaves `Accept-Encoding` to the adapter; see `Flow.build_http_client`.
         def on_data
           proc do |chunk, _received_bytes, _env|
             @buffer << chunk

lib/mcp/client/oauth/client_credentials_provider.rb

@@ -46,6 +46,8 @@ module OAuth
       #   server requires beyond the grant itself (Auth0's `audience`, for example).
       #   A key in `Flow::RESERVED_TOKEN_REQUEST_PARAMS` raises `Flow::InvalidTokenRequestParamsError`.
       #   The Hash is copied and frozen. See `StorageBackedProvider#token_request_params`.
+      # - `http_client_customizer` - Callable invoked with the `Faraday::Connection` the flow builds for
+      #   its own requests; see `StorageBackedProvider#http_client_customizer`.
       class ClientCredentialsProvider
         include StorageBackedProvider
 
@@ -66,7 +68,8 @@ def initialize(
           scope: nil,
           storage: nil,
           authorization_request_validator: nil,
-          token_request_params: nil
+          token_request_params: nil,
+          http_client_customizer: nil
         )
           if blank?(client_id)
             raise InvalidCredentialsError, "client_id is required for the client_credentials grant."
@@ -104,13 +107,16 @@ def initialize(
             client_information["client_secret"] = client_secret
           end
 
+          http_client_customizer = validated_http_client_customizer(http_client_customizer)
+
           @client_id = client_id
           @private_key = private_key
           @signing_algorithm = signing_algorithm
           @scope = scope
           @storage = storage || InMemoryStorage.new
           @authorization_request_validator = authorization_request_validator
           @token_request_params = frozen_token_request_params(token_request_params)
+          @http_client_customizer = http_client_customizer
           @storage.save_client_information(client_information)
         end
 

lib/mcp/client/oauth/cross_app_access_provider.rb

@@ -29,6 +29,8 @@ module OAuth
       #   the authorization server requires beyond the grant itself. A key in `Flow::RESERVED_TOKEN_REQUEST_PARAMS` raises
       #   `Flow::InvalidTokenRequestParamsError`. The Hash is copied and frozen.
       #   See `StorageBackedProvider#token_request_params`.
+      # - `http_client_customizer` - Callable invoked with the `Faraday::Connection` the flow builds for its own requests;
+      #   see `StorageBackedProvider#http_client_customizer`.
       #
       # https://github.com/modelcontextprotocol/modelcontextprotocol/issues/990
       class CrossAppAccessProvider
@@ -46,7 +48,8 @@ def initialize(
           scope: nil,
           storage: nil,
           authorization_request_validator: nil,
-          token_request_params: nil
+          token_request_params: nil,
+          http_client_customizer: nil
         )
           if blank?(client_id)
             raise InvalidConfigurationError, "client_id is required for the jwt-bearer grant."
@@ -60,11 +63,14 @@ def initialize(
             raise InvalidConfigurationError, "assertion_provider must be callable as `call(audience:, resource:)` and return the ID-JAG assertion."
           end
 
+          http_client_customizer = validated_http_client_customizer(http_client_customizer)
+
           @assertion_provider = assertion_provider
           @scope = scope
           @storage = storage || InMemoryStorage.new
           @authorization_request_validator = authorization_request_validator
           @token_request_params = frozen_token_request_params(token_request_params)
+          @http_client_customizer = http_client_customizer
           @storage.save_client_information(
             "client_id" => client_id,
             "client_secret" => client_secret,

lib/mcp/client/oauth/flow.rb

@@ -63,6 +63,71 @@ class AuthorizationRefusedError < AuthorizationError; end
         # a failed refresh as a reason to run the interactive flow.
         class InvalidTokenRequestParamsError < ArgumentError; end
 
+        # Raised by `RequestedOriginGuard` when middleware added through the provider's `http_client_customizer`
+        # would send a request to an origin other than the one the flow validated, or has dropped the record of
+        # the URL the flow asked for. An `ArgumentError` because the middleware is a configuration mistake,
+        # and deliberately outside `AuthorizationError`, which discovery treats as "nothing published"
+        # and `MCP::Client::HTTP` treats on a failed refresh as a reason to run the interactive flow.
+        class DestinationMismatchError < ArgumentError; end
+
+        # Faraday middleware registered on the connection `build_http_client` assembles before the customizer
+        # is invoked, so with the usual `use` it sits ahead of the customizer's middleware and sees the URL exactly
+        # as the flow requested it, which it records on the request environment for `RequestedOriginGuard`.
+        # The guard covers what happens to a request after that record; middleware inserted ahead of it with
+        # `builder.insert(0, ...)` that rewrites the URL before it or rebuilds the environment is outside the guard.
+        # The record lives on the environment, not in `env.request.context`: that slot belongs to the application,
+        # which may fill it on the connection or replace it from a middleware of its own. Only the first URL seen
+        # on an environment is recorded: a middleware inserted ahead of this one that re-enters the stack after
+        # a `3xx` with the same environment, or with its `dup`, which shares the record, cannot replace it with
+        # the redirected URL.
+        class RequestedURLStamp
+          KEY = :mcp_oauth_requested_url
+
+          def initialize(app)
+            @app = app
+          end
+
+          def call(env)
+            env[KEY] ||= env.url.to_s
+            @app.call(env)
+          end
+        end
+
+        # Faraday middleware registered last on that connection, so it sees `env.url` after any customizer-added
+        # middleware has rewritten it or followed a redirect. A request that would leave the origin the flow asked
+        # for is refused before it reaches the adapter, since every destination check ran against the URL as
+        # written; a same-origin change stays with the server those checks admitted. The record survives
+        # the `env.dup` that redirect-following middleware performs, and a request that arrives without it is
+        # refused as well, so a middleware that rebuilds the environment fails closed rather than open.
+        # The origin boundary resembles the one the Python SDK keeps for its own auth requests, which follows
+        # a redirect itself only within the origin; this flow follows none.
+        class RequestedOriginGuard
+          def initialize(app)
+            @app = app
+          end
+
+          def call(env)
+            requested = env[RequestedURLStamp::KEY]
+            unless requested
+              raise DestinationMismatchError, <<~MESSAGE
+                Request to #{Discovery.canonicalize_origin_and_path(env.url.to_s).inspect} carries no record of \
+                the URL the flow asked for; middleware that rebuilds the request environment is refused.
+              MESSAGE
+            end
+
+            unless Discovery.same_origin?(env.url.to_s, requested)
+              raise DestinationMismatchError, <<~MESSAGE
+                Request to #{Discovery.canonicalize_origin_and_path(requested).inspect} would be sent to \
+                #{Discovery.canonicalize_origin_and_path(env.url.to_s).inspect}, on a different origin; \
+                middleware that follows redirects or rewrites URLs is refused.
+              MESSAGE
+            end
+
+            @app.call(env)
+          end
+        end
+        private_constant :RequestedURLStamp, :RequestedOriginGuard
+
         class << self
           # Returns why `params` cannot ride a token request as `token_request_params`, or `nil` when it can.
           # Shared by the provider constructors and the flow, which both refuse the value with `InvalidTokenRequestParamsError`,
@@ -82,6 +147,33 @@ def token_request_params_problem(params)
 
             nil
           end
+
+          # Builds the connection the flow uses for its own requests: the SDK's defaults, `RequestedURLStamp`,
+          # then `customizer` (a provider's `http_client_customizer`, called with the `Faraday::Connection`),
+          # then `RequestedOriginGuard` last so it sees what the customizer's middleware does to each request
+          # after the stamp recorded it. Every request on the connection passes through both, so a caller using
+          # it directly is held to the same origin rule.
+          #
+          # Deliberately built without redirect-following middleware. Every destination check in this class runs
+          # against the URL as written, before the request goes out, so a connection that transparently followed
+          # a `3xx` would let a server reach a host the checks just refused. The guard turns following at
+          # the middleware level into a refusal; following inside an adapter stays invisible, so a customizer must
+          # not enable it.
+          #
+          # `Accept-Encoding` is deliberately left unset. `Net::HTTP::GenericRequest` negotiates it and decodes
+          # the response only while the caller has not claimed that header; assigning it turns `decode_content` off,
+          # which would silently move `BoundedBody`'s cap onto compressed bytes and let a small body expand past it
+          # after the check.
+          def build_http_client(customizer = nil)
+            require "faraday"
+
+            Faraday.new do |faraday|
+              faraday.headers["Accept"] = "application/json"
+              faraday.use(RequestedURLStamp)
+              customizer&.call(faraday)
+              faraday.use(RequestedOriginGuard)
+            end
+          end
         end
 
         def initialize(provider:, http_client_factory: nil)
@@ -1298,22 +1390,18 @@ def http_client
           @http_client ||= @http_client_factory.call
         end
 
-        # Deliberately built without redirect-following middleware. Every destination check in
-        # this class runs against the URL as written, before the request goes out, so a connection
-        # that transparently followed a `3xx` would let a server reach a host the checks just refused.
-        # A caller passing `http_client_factory:` takes on that responsibility: add redirect following here
-        # and the guards above only cover the first hop.
-        #
-        # `Accept-Encoding` is deliberately left unset. `Net::HTTP::GenericRequest` negotiates it and decodes
-        # the response only while the caller has not claimed that header; assigning it turns `decode_content` off,
-        # which would silently move `BoundedBody`'s cap onto compressed bytes and let a small body expand past it
-        # after the check.
+        # A connection supplied through `http_client_factory:` replaces this one, the provider's customizer and
+        # `RequestedOriginGuard` included, so that caller takes on the redirect responsibility described on
+        # `build_http_client`; `bounded_request` caps its responses all the same.
         def default_http_client
-          require "faraday"
+          self.class.build_http_client(provider_http_client_customizer)
+        end
 
-          Faraday.new do |faraday|
-            faraday.headers["Accept"] = "application/json"
-          end
+        # `nil` for a provider that predates the hook or leaves it unset.
+        def provider_http_client_customizer
+          return unless @provider.respond_to?(:http_client_customizer)
+
+          @provider.http_client_customizer
         end
 
         def response_body_string(response)