Add OAuth 2.1 auth flow and per-request client support for HTTP transport

[yonglingsong]

Summary

• Add OAuth 2.1 proxy endpoints (RFC 8414/9728) so MCP clients (Claude Code, Cursor, etc.) can authenticate via any upstream identity provider through standard OAuth discovery
• Add per-request DataHubClient creation from Bearer tokens in HTTP headers, with proper lifecycle management (created per request, closed after)
• Add pluggable TokenValidator protocol for validating/exchanging Bearer tokens before creating DataHub clients (e.g., OIDC validation, token exchange)
• Add generic OIDCTokenValidator for OSS deployments using any OIDC-compliant provider
• Add BearerTokenDetectionMiddleware (ASGI layer) that returns HTTP 401 with WWW-Authenticate header to trigger MCP client OAuth flows
• Add integration tests for multi-threaded client isolation, fallback token behavior, and OAuth endpoint correctness
• Update DEVELOPING.md with step-by-step instructions for running auth flow integration tests

Key design decisions

• Pluggable token validation: The TOKEN_VALIDATOR_FACTORY env var points to a module:callable that returns a TokenValidator instance. This keeps vendor-specific auth (Apple Notary, custom JWT exchange) out of the
  core server.
• Test tokens via env vars only: Integration tests take tokens via MCP_TEST_AUTH_TOKEN / MCP_TEST_AUTH_TOKEN_2 — no CLI tool dependencies in the test setup. This keeps test files identical across OSS and internal repos.

---

Note

High Risk
Adds new OAuth 2.1 discovery/proxy endpoints and changes HTTP request authentication to create/validate per-request DataHubClients, which directly impacts auth enforcement and token handling. Misconfiguration or bugs could lead to unintended 401s, token bypass, or resource leaks under concurrency.

Overview
Enables OAuth 2.1 auth discovery and login flows for MCP clients by adding /.well-known/* metadata plus oauth/* proxy routes that forward authorize/token exchanges to an upstream IdP, including a short-lived in-memory state store.

Reworks HTTP auth handling so each request can create a fresh DataHubClient from the Authorization: Bearer header (optionally via a pluggable TOKEN_VALIDATOR_FACTORY), falls back to an env-based startup client when available, and closes per-request clients after each call; an ASGI BearerTokenDetectionMiddleware returns an HTTP 401 with WWW-Authenticate to trigger client OAuth when OAuth is enabled and no fallback token exists.

Adds httpx dependency, a new generic OIDCTokenValidator, new unit/integration tests covering OAuth endpoints, multi-token concurrency/isolation, and fallback-token behavior, and updates DEVELOPING.md with revised test instructions and required env vars.

^{Written by Cursor Bugbot for commit 708b318. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

You have used all of your free Bugbot PR reviews.

To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.