[improve][broker] PIP-490: Zero-downtime token signing-key rotation for the built-in JWT provider

pip/pip-490.md

@@ -0,0 +1,183 @@
+# PIP-490: Zero-downtime token signing-key rotation for the built-in JWT provider
+
+# Background knowledge
+
+`AuthenticationProviderToken` (in `pulsar-broker-common`) authenticates clients that present a signed JWT. Today it is configured with one validation key: either a symmetric secret (`tokenSecretKey`) or an asymmetric public key (`tokenPublicKey`, whose algorithm is set by `tokenPublicAlg`). It checks every incoming JWT against that one key. The token's `sub` claim (or a configured `tokenAuthClaim`) becomes the client's role. The binary listener, the HTTP/HTTPS and WebSocket endpoints, and the Pulsar Proxy all authenticate tokens through this provider. It uses the jjwt library (`io.jsonwebtoken`), version 0.13.
+
+A JWT can carry a `kid` ("key id") in its header that names the key used to sign it. Systems that hold several signing keys at once use the `kid` to pick the right key to verify with. This is what makes smooth rotation possible: you add a new key, start signing new tokens with it, and keep accepting both the old and the new key until the old tokens expire. A JWKS (JSON Web Key Set) is a JSON document that holds a list of keys; each entry states its own `kid`, its key type (`kty`), and, optionally, its algorithm (`alg`). That makes a JWKS a natural way to describe "a set of validation keys".
+
+Pulsar already resolves keys by `kid`, but only in `AuthenticationProviderOpenID`. That provider is built on a  different library (`com.auth0.jwt`) and expects to fetch keys from a remote `jwks_uri`. So this proposal borrows the *idea*, not the code. The two things it needs are already in the jjwt 0.13 library that the token provider depends on: `keyLocator`, which lets the parser pick a key per token from the header, and a local JWKS parser (`io.jsonwebtoken.security.Jwks` / `JwkSet`). For comparison, Kafka's OAUTHBEARER path also uses a JWKS and treats online key rotation as normal.
+
+# Motivation
+
+The provider loads one key when it starts, and never looks at the token's `kid` (`AuthenticationProviderToken.java`):
+
+```java
+private Key validationKey;
+this.validationKey = getValidationKey(config);
+this.parser = Jwts.parser()
+        .setAllowedClockSkewSeconds(allowedSkew)
+        .setSigningKey(this.validationKey).build();
+```
+
+It also has no way to reload the key: the key stays fixed for the life of the process. Because of this, you cannot rotate a signing key without downtime. There is no period when the old key and the new key are both accepted. To rotate, an operator has to re-issue every existing token with the new key and switch the broker's config at the same moment (in practice, a coordinated restart). During the switch, tokens signed with the wrong" key are rejected.
+
+This makes two normal security tasks painful:
+
+- **Responding to a possible key compromise.** The safe move is to add a replacement key and retire the old one gradually. Today that means an outage plus re-issuing every token at once.
+- **Routine rotation.** Rotating keys on a schedule is a basic control (see NIST SP 800-57 on cryptoperiods). Today it is expensive enough that operators tend to skip it and keep long-lived keys.
+
+Graceful rotation is also a building block for any future work on broker-issued, revocable delegation tokens. The fix needs no new cryptography and no new library. It only changes how the provider validates tokens: it adds `kid`-based selection, support for several keys, and reloading without a restart. It does this carefully, so that supporting several keys does not open the "algorithm confusion" weakness that a naive "just try every key" approach would (explained under Security Considerations).
+
+# Goals
+
+## In Scope
+
+- Let `AuthenticationProviderToken` hold several validation keys at once. Each key is tied to a `kid` and to one algorithm.
+- Choose the key using the token's `kid`. If a token has no `kid` (older tokens), fall back to trying the configured keys, each with its own algorithm.
+- Reload the set of keys without restarting the broker.
+- Keep today's behavior exactly the same when only the existing single-key settings are used.
+- Work everywhere the provider is used: the binary protocol, HTTP/HTTPS, WebSocket, and the Pulsar Proxy.
+
+## Out of Scope
+
+- Broker-issued / delegation tokens and revocation. That is possible future work, and this proposal is a step toward it.
+- Changing the JWT library, or adding algorithms that jjwt does not already support.
+- Generating or distributing keys, or fetching keys from a remote `jwks_uri`. That is what the OIDC provider is for. Keys stay local and operator-supplied.
+- Claim handling (`exp`, `tokenAuthClaim`, `tokenAudienceClaim`). Only the choice of *which key* verifies the signature changes.
+
+# High Level Design
+
+Replace the single `validationKey` with a read-only holder that maps each `kid` to a key and its algorithm. The provider looks up the key for each token, and can replace the whole holder in one step when the keys are reloaded.
+
+The main way to supply several keys is a local JWKS document (a file, or inline text). A JWKS is a good fit because each key already states its `kid` and key type, and can also state an exact `alg`. That is exactly the information needed to tie each key to the one algorithm it is allowed to verify - which is the heart of the security design. The existing `tokenSecretKey` / `tokenPublicKey` settings still work; such a key is treated as a single key with no `kid`.
+
+How a token is resolved:
+
+- If the token has a `kid` that matches a configured key, verify it against that key only, using that key's algorithm.
+- If the token has no `kid` (or its `kid` matches nothing, and fallback is on), try each configured key in turn, each with its own algorithm. The first key that verifies wins.
+- If no key verifies, the token is rejected, just as today (`INVALID_TOKEN`), with an error code that shows whether the `kid` was the problem.
+
+The holder is kept in a `volatile` field and replaced as a whole on reload, so a request never sees a half-updated set. The jjwt parser is still built once; its `keyLocator` reads the current holder for each token.
+
+The rotation steps are all additive, and no valid token is ever rejected in flight:
+
+1. Add key **B** to the JWKS. Both A and B are now accepted.
+2. Start issuing new tokens signed with **B**, stamped with `kid=B`.
+3. Wait until all tokens signed with **A** have expired.
+4. Remove key **A**.
+
+Stamping a `kid` on new tokens already works today: `AuthTokenUtils.createToken` passes through a headers map, and `pulsar tokens create --headers kid=B` already sets it. This proposal only changes the validation side, and adds an optional `--key-id` flag for convenience.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Choosing the key
+
+Replace the single `Key validationKey` with a read-only holder - call it `TokenValidationKeys` - that maps each `kid` to its key and algorithm, plus the ordered list of keys that older, `kid`-less tokens may be tried against. Keep the holder in a `volatile` field so a reload can replace it in one step. Wire it into the parser once, using jjwt's key locator instead of `setSigningKey`:
+
+- **When a token has a `kid` (one lookup).** Build the parser with `Jwts.parser().keyLocator(locator)`. For each token, `locator.locate(header)` reads `header.getKeyId()` and `header.getAlgorithm()` from the (still unverified) `JwsHeader` - both methods exist in jjwt 0.13 - and returns the key for that `kid`, or `null` (in which case jjwt reports the usual verification failure).
+- **The algorithm is checked in the locator, not left to jjwt's defaults.** `locate` returns the key only if the token's `alg` matches the key's algorithm; otherwise it returns `null`. Because the keys come from a JWKS, each key already has its algorithm attached, so this match is checked for you. This is what stops the RS256-to-HS256 confusion attack (see Security Considerations).
+- **When a token has no `kid` (older tokens).** jjwt's locator picks one key per parse, so it cannot "try them all" on its own. The fallback is therefore a short loop in the provider: try to verify against each candidate key in turn, each with its own algorithm (an HMAC key only runs HMAC, an RSA key only runs RSA), and stop at the first that works. This loop is the cost referred to in the DoS note below. Setting `tokenRequireKid=true`removes the loop entirely (strict matching only).
+
+Claim handling (`exp`, `tokenAuthClaim`, `tokenAudienceClaim`) does not change.
+
+### Loading and reloading
+
+`getValidationKeys(config)` builds the holder from two possible sources, which can be combined: the existing `tokenSecretKey` / `tokenPublicKey` (one key, no `kid`, with the algorithm from `tokenPublicAlg`, or HMAC for a secret), and the new `tokenValidationKeysJwks` source, parsed with jjwt's `Jwks` / `JwkSet`. A scheduled task(`tokenValidationKeysRefreshSeconds`; `0` means load once, as today) re-reads the source, builds a new read-only holder, and swaps it in. It only reads local files; it never makes a network call. If the source fails to parse, the provider keeps the previous holder and logs and counts the failure, so a bad edit never  leaves the broker with no keys.
+
+To stay exactly compatible: if only the old setting is present and `tokenValidationKeysJwks` is empty, the provider uses the original single-key path (the parser is built with that one key), so behavior is identical to today.
+
+The reload timer is a single-thread `ScheduledExecutorService`. The provider creates it only when `tokenValidationKeysRefreshSeconds > 0`, and shuts it down in `close()` (which does nothing today).
+
+### Creating tokens
+
+Nothing new is needed to set a `kid`: `createToken` already passes headers to `JwtBuilder.setHeaderParams`, and the CLI already accepts `--headers key=value`. This proposal adds an optional `--key-id` flag to `pulsar tokens create` for convenience; it validates the value (rejecting an empty or duplicate `kid`).
+
+## Public-facing Changes
+
+### Public API
+
+None. No REST change, no client-API change, and no change to the `AuthenticationProvider` interface.
+
+### Binary protocol
+
+None. The `kid` is a standard JWT header and does not affect the Pulsar protocol.
+
+### Configuration
+
+New settings on `ServiceConfiguration` (`broker.conf`). They must be added to `ProxyConfiguration` (`proxy.conf`) too, with the same names. The proxy builds its `AuthenticationService` from a `ServiceConfiguration` created by `PulsarConfigurationLoader.convertFrom(ProxyConfiguration)`, which copies fields by matching their names. If a setting exists only on `ServiceConfiguration`, the proxy would validate against a different key than the broker. An integration test should go through the proxy.
+
+- `tokenValidationKeysJwks` - a local JWKS source (`file://`, `data:`, or inline base64, using the same loader as the single-key settings). It holds one or more keys, each with a `kid` and key type, and optionally an exact `alg`. Empty by default.
+- `tokenValidationKeysRefreshSeconds` - how often to reload the source. `0` (the default) means load once.
+- `tokenRequireKid` - `false` by default. When `true`, a token whose `kid` does not match a configured key is rejected instead of going through the fallback.
+
+The existing `tokenSecretKey` / `tokenPublicKey` / `tokenPublicAlg` still work and combine with the set (as one unnamed key). Like the other token settings, the three new ones respect `tokenSettingPrefix`, so a prefixed or chained provider picks them up.
+
+### CLI
+
+An optional `--key-id` flag on `pulsar tokens create` sets the JWT `kid` header. It is a validated shortcut for `--headers kid=...`.
+
+### Metrics
+
+Reuse `AuthenticationMetricsToken`. Add one value to the `ErrorCode` enum (today `INVALID_AUTH_DATA`, `INVALID_TOKEN`, `INVALID_AUDIENCES`):
+
+- `INVALID_TOKEN_KID` (new) - the token named a `kid` that matches no configured key (either with `tokenRequireKid=true`, or after the fallback also failed).
+
+Also add a reload-health signal - a counter of reload failures and a gauge of how many keys are configured - using the provider's existing OpenTelemetry naming. This lets an operator see a broken JWKS without reading logs.
+
+# Monitoring
+
+During a rotation, watch the per-error-code failure counter and the reload health:
+
+- If `INVALID_TOKEN_KID` is above zero after you add key B but before clients have the new tokens, the set is missing a key that clients are already sending. Add it before going further.
+- Before step 4 (removing key A), confirm that successful logins using key A have dropped to zero - that is, all A-signed tokens have expired.
+- If the reload-failure count rises, the JWKS on disk no longer parses. The broker keeps serving the last good set, but your change has not taken effect.
+
+# Security Considerations
+
+- **Algorithm confusion is the main risk this design has to close, and does.** The classic JWT attack forges a token with `alg=HS256`, using the bytes of a public RSA key as the HMAC secret. A single-key provider is mostly safe because it has one key of one type. A multi-key provider that "tries every key" is not safe, unless each key is tied to one algorithm. Here, a key is only ever used with its own algorithm, and a token whose `alg` does not match is rejected before any signature check. Tying the key type alone is already enough to stop the RSA-to-HMAC case: a public RSA key is a `PublicKey` and can never be used as an HMAC `SecretKey`. An exact `alg` on the key narrows it further. Using a JWKS makes this automatic, because each key states its type (and optionally its `alg`), so the binding is checked by the code rather than remembered by the operator. Mixing symmetric and asymmetric keys is therefore safe. The docs should still advise a separate`kid` for each algorithm, and never reusing a `kid` across key types.
+- **The `kid` comes from the attacker, and that is fine.** It only picks which key to try. The signature still has to verify against that key, using that key's algorithm. The algorithm binding and the operator-chosen key set stop a `kid` from steering the check toward a weaker key.
+- **Fallback versus strict matching.** The fallback trades a little strictness for easier migration. Once every token has a `kid`, set `tokenRequireKid=true` to require an exact match (recommended after migration).
+- **Cost and denial of service.** The fallback does up to N signature checks per unverified token, and public-key checks are not cheap. A rotation only needs two or three keys, so keep the set small; the docs should say so, and the code should log a warning if the set grows large. `tokenRequireKid` limits this to one check per token.
+- **No new trust, and no network calls.** Every key is supplied locally by the operator, as today. Unlike OIDC, the provider never fetches a `jwks_uri`.
+- **Safe reloads.** A failed reload keeps the last good set, so the broker never drops to zero keys. The swap is atomic, so no request is checked against a half-built set.
+- **Multi-tenancy is unchanged.** Only signature checking changes.
+
+# Backward & Forward Compatibility
+
+## Upgrade
+
+Nothing to do. If only `tokenSecretKey` / `tokenPublicKey` are set and tokens have no `kid`, the provider uses the original single-key path and behaves exactly as before. Multiple keys are opt-in through `tokenValidationKeysJwks`.
+
+## Downgrade / Rollback
+
+Remove the new settings, or downgrade the broker and proxy. The provider goes back to single-key validation. One caution: before rolling back, make sure every live token is signed with the one key the old broker keeps. Tokens signed with any other key in the set will stop validating. The safe order is the reverse of rotation: re-issue tokens with the key you are keeping, wait for the others to expire, then downgrade.
+
+## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations
+
+Keys are per-broker and per-proxy configuration. Brokers do not share them, and geo-replication copies data, not login sessions. When rotating a key used across clusters, add the new key to every cluster's set before any cluster starts using it, and remove the old key everywhere only after all old tokens have expired everywhere. Keep brokers and proxies that may serve the same clients on the same set of keys.
+
+# Alternatives
+
+- **A plain `kid:file://...` list instead of a JWKS.** Rejected as the main format. A bare key file does not state its algorithm, so you would have to declare the `kid`-to-algorithm mapping separately, and any mistake there reopens the algorithm-confusion hole. A JWKS keeps the `kid`, key type, and `alg` together, is parsed by the library already in use, and lets the code check the binding. (The `data:` / `file:` loader is still reused, to load the JWKS document.)
+- **Reuse the OIDC provider for local tokens.** Rejected. It expects an HTTP issuer with a remote `jwks_uri`, and uses a different library. Local token auth is deliberately free of network calls; only the `kid`-selection idea is borrowed.
+- **Chain two token providers with `AuthenticationProviderList`.** Possible today, but rejected as the path forward: it doubles the work, makes metrics and failure attribution harder, cannot select by `kid`, and cannot hot-reload.
+- **A remote `jwks_uri` for the local provider.** Out of scope - that is essentially the OIDC provider.
+
+# General Notes
+
+The change is small and contained: a read-only map of `kid` to key and algorithm, a `keyLocator` that ties each key to one algorithm, a timed atomic reload, three new settings mirrored into `ProxyConfiguration`, and one new error code. Both building blocks - `keyLocator` and the local `Jwks` parser - are already in the jjwt 0.13 library. The benefit is concrete: graceful key rotation and a real response to key compromise, with no downtime, and without opening the algorithm-confusion weakness that a naive multi-key design would.
+
+How this relates to nearby proposals: **PIP-488** (authentication audit events) works well with this: an `INVALID_TOKEN_KID` failure would show up as an `AuthenticationEvent`, giving a per-login view of a rotation in progress. **PIP-489** (FIPS mode) is unrelated here; its dual-verify HMAC change for the SASL role-token signer is a similar "accept two keys during migration" idea, but for a different key.
+
+# Links
+
+* jjwt 0.13 - `JwtParserBuilder.keyLocator(Locator<Key>)` and `io.jsonwebtoken.security.Jwks` / `JwkSet`.
+* RFC 7517 - JSON Web Key (JWK) and JWK Set: https://www.rfc-editor.org/rfc/rfc7517
+* RFC 7515 §4.1.4 - the JWS `kid` header: https://www.rfc-editor.org/rfc/rfc7515#section-4.1.4
+* NIST SP 800-57 Part 1 - key management / cryptoperiods: https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final
+* Related: a future broker-issued delegation-token PIP (this is a prerequisite).
+* Mailing List discussion thread: TBD
+* Mailing List voting thread: TBD