|
| 1 | +--- |
| 2 | +title: external |
| 3 | +title_tag: external Pulumi ESC provider |
| 4 | +meta_desc: The `external` Pulumi ESC provider enables you to dynamically import values from custom secret sources. |
| 5 | +h1: external |
| 6 | +menu: |
| 7 | + esc: |
| 8 | + identifier: external |
| 9 | + parent: esc-dynamic-secrets |
| 10 | +--- |
| 11 | + |
| 12 | +# External Provider |
| 13 | + |
| 14 | +The `external` provider enables you to integrate custom secret sources with Pulumi ESC by making authenticated HTTP requests to user-controlled adapter services. |
| 15 | + |
| 16 | +## Overview |
| 17 | + |
| 18 | +The external provider serves as a generic escape hatch for integrating secret sources that don't have native Pulumi ESC support. Instead of waiting for a native provider implementation, you can build a custom HTTPS adapter service that: |
| 19 | + |
| 20 | +- Authenticates requests using JWT tokens issued by Pulumi Cloud |
| 21 | +- Receives configuration from your ESC environment |
| 22 | +- Returns secrets back to ESC |
| 23 | + |
| 24 | +## When to Use |
| 25 | + |
| 26 | +Use the external provider when: |
| 27 | + |
| 28 | +- You need to integrate a custom or proprietary secret management system |
| 29 | +- You have specific business logic for secret fetching |
| 30 | +- Your secret source is behind a firewall or requires custom networking |
| 31 | + |
| 32 | +## ESC Configuration Example |
| 33 | + |
| 34 | +```yaml |
| 35 | +values: |
| 36 | + customSecrets: |
| 37 | + fn::open::external: |
| 38 | + url: https://my-adapter.example.com/fetch-secrets |
| 39 | + request: |
| 40 | + environment: production |
| 41 | + secretType: api-keys |
| 42 | + secret: true # Optional, defaults to true |
| 43 | +``` |
| 44 | +
|
| 45 | +### Request Payload |
| 46 | +
|
| 47 | +Your adapter receives a POST request with the `request` field from your ESC configuration: |
| 48 | + |
| 49 | +```json |
| 50 | +{ |
| 51 | + "environment": "production", |
| 52 | + "secretType": "api-keys" |
| 53 | +} |
| 54 | +``` |
| 55 | + |
| 56 | +### Response Payload |
| 57 | + |
| 58 | +Your adapter returns a JSON object that becomes available under the `response` key: |
| 59 | + |
| 60 | +```json |
| 61 | +{ |
| 62 | + "apiKey": "secret-api-key-value", |
| 63 | + "apiSecret": "secret-api-secret-value", |
| 64 | + "endpoint": "https://api.example.com" |
| 65 | +} |
| 66 | +``` |
| 67 | + |
| 68 | +In ESC, you should see output like the following: |
| 69 | + |
| 70 | +```json |
| 71 | +{ |
| 72 | + "customSecrets": { |
| 73 | + "response": { |
| 74 | + "apiKey": "secret-api-key-value", |
| 75 | + "apiSecret": "secret-api-secret-value", |
| 76 | + "endpoint": "https://api.example.com" |
| 77 | + } |
| 78 | + } |
| 79 | +} |
| 80 | +``` |
| 81 | + |
| 82 | +You can mark the entire response as secret with `secret: true` (the default). |
| 83 | + |
| 84 | +## Building Custom Adapters |
| 85 | + |
| 86 | +### Requirements |
| 87 | + |
| 88 | +Your adapter service must: |
| 89 | + |
| 90 | +1. **Run on HTTPS** - Pulumi ESC only makes requests to `https://` URLs |
| 91 | +2. **Accept POST requests** with `Content-Type: application/json` |
| 92 | +3. **Validate JWT tokens** from the `Authorization: Bearer <token>` header |
| 93 | +4. **Return JSON responses** with `Content-Type: application/json` |
| 94 | +5. **Return HTTP 200** for successful requests (other status codes are treated as errors) |
| 95 | + |
| 96 | +### JWT Authentication |
| 97 | + |
| 98 | +Every request includes a JWT token in the `Authorization` header. The token is signed using RS256 and can be verified using Pulumi Cloud's public JWKS. |
| 99 | + |
| 100 | +The JWT token includes the following claims, which you can use to make authorization decisions: |
| 101 | + |
| 102 | +| Claim | Description | Example | |
| 103 | +|----------------|---------------------------------------------------|-------------------------------------------------------| |
| 104 | +| `iss` | Issuer (Pulumi Cloud URL) | `https://api.pulumi.com` | |
| 105 | +| `sub` | Subject (environment identity) | `pulumi:environments:org:acme-corp:env:prod` | |
| 106 | +| `aud` | Audience (your adapter URL) | `https://my-adapter.example.com/fetch-secrets` | |
| 107 | +| `exp` | Expiration time (Unix timestamp) | `1736937600` | |
| 108 | +| `iat` | Issued at (Unix timestamp) | `1736933600` | |
| 109 | +| `jti` | Unique id (to prevent replay) | `a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11` | |
| 110 | +| `org` | Pulumi organization name | `acme-corp` | |
| 111 | +| `env` | Environment name (legacy format) | `prod` | |
| 112 | +| `current_env` | Current environment (fully qualified) | `acme-corp/prod` | |
| 113 | +| `root_env` | Root environment in import chain | `acme-corp/base` | |
| 114 | +| `trigger_user` | User who opened the environment | `alice` | |
| 115 | +| `body_hash` | Hash of request body (for integrity) | `sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=` | |
| 116 | + |
| 117 | +#### Validating Requests |
| 118 | + |
| 119 | +Your adapter should: |
| 120 | + |
| 121 | +1. **Extract the JWT** from the `Authorization: Bearer <token>` header |
| 122 | +2. **Verify the signature** using the public key from JWKS |
| 123 | +3. **Validate standard claims**: |
| 124 | + - `aud` matches your adapter URL |
| 125 | + - `exp` has not passed (token not expired) |
| 126 | + - `iss` is your Pulumi Cloud instance |
| 127 | +4. **Verify body integrity** by generating an [SRI hash](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) of the request body: |
| 128 | + - Compute SHA-256 hash of request body |
| 129 | + - Base64-encode the hash |
| 130 | + - Verify it matches the `body_hash` claim with `sha256-` prefix |
| 131 | + |
| 132 | +The `body_hash` claim binds the JWT to the request body to prevent replay attacks where an attacker could reuse a valid JWT with a different request body. |
| 133 | + |
| 134 | +### Example Adapter Implementation |
| 135 | + |
| 136 | +Here's a complete adapter in Python that fetches secrets from environment variables: |
| 137 | + |
| 138 | +```python |
| 139 | +#!/usr/bin/env python3 |
| 140 | +""" |
| 141 | +Example external provider adapter for Pulumi ESC. |
| 142 | +Fetches secrets from environment variables. |
| 143 | +""" |
| 144 | +
|
| 145 | +import hashlib |
| 146 | +import base64 |
| 147 | +import json |
| 148 | +import os |
| 149 | +from http.server import HTTPServer, BaseHTTPRequestHandler |
| 150 | +
|
| 151 | +import jwt |
| 152 | +from jwt import PyJWKClient |
| 153 | +
|
| 154 | +# Configuration |
| 155 | +JWKS_URL = "https://api.pulumi.com/.well-known/jwks.json" |
| 156 | +ADAPTER_URL = "https://my-adapter.example.com/fetch-secrets" |
| 157 | +PORT = 8443 |
| 158 | +
|
| 159 | +# Initialize JWKS client (caches keys automatically) |
| 160 | +jwks_client = PyJWKClient(JWKS_URL) |
| 161 | +
|
| 162 | +
|
| 163 | +def verify_body_hash(body: bytes, claims: dict) -> None: |
| 164 | + """Verify the body_hash claim matches the request body.""" |
| 165 | + expected_hash = claims.get("body_hash") |
| 166 | + if not expected_hash: |
| 167 | + raise ValueError("Missing body_hash claim") |
| 168 | +
|
| 169 | + # Compute SHA-256 hash in SRI format |
| 170 | + hash_digest = hashlib.sha256(body).digest() |
| 171 | + actual_hash = f"sha256-{base64.b64encode(hash_digest).decode('ascii')}" |
| 172 | +
|
| 173 | + if actual_hash != expected_hash: |
| 174 | + raise ValueError(f"Body hash mismatch: expected {expected_hash}, got {actual_hash}") |
| 175 | +
|
| 176 | +
|
| 177 | +class AdapterHandler(BaseHTTPRequestHandler): |
| 178 | + def do_POST(self): |
| 179 | + try: |
| 180 | + # Extract token from Authorization header |
| 181 | + auth_header = self.headers.get("Authorization", "") |
| 182 | + if not auth_header.startswith("Bearer "): |
| 183 | + self.send_error(401, "Missing or invalid Authorization header") |
| 184 | + return |
| 185 | +
|
| 186 | + token = auth_header[7:] # Remove "Bearer " prefix |
| 187 | +
|
| 188 | + # Get signing key from JWKS and verify token |
| 189 | + signing_key = jwks_client.get_signing_key_from_jwt(token) |
| 190 | + claims = jwt.decode( |
| 191 | + token, |
| 192 | + signing_key.key, |
| 193 | + algorithms=["RS256"], |
| 194 | + audience=ADAPTER_URL, |
| 195 | + options={"verify_exp": True} |
| 196 | + ) |
| 197 | +
|
| 198 | + # Read and verify request body |
| 199 | + content_length = int(self.headers.get("Content-Length", 0)) |
| 200 | + body = self.rfile.read(content_length) |
| 201 | + verify_body_hash(body, claims) |
| 202 | +
|
| 203 | + # Parse request |
| 204 | + request = json.loads(body) |
| 205 | + secret_name = request.get("secretName") |
| 206 | +
|
| 207 | + if not secret_name: |
| 208 | + self.send_error(400, "Missing required field: secretName") |
| 209 | + return |
| 210 | +
|
| 211 | + # Fetch secret from environment variable |
| 212 | + secret_value = os.environ.get(secret_name) |
| 213 | + if not secret_value: |
| 214 | + self.send_error(404, f"Secret not found: {secret_name}") |
| 215 | + return |
| 216 | +
|
| 217 | + # Return response |
| 218 | + response = {"value": secret_value} |
| 219 | +
|
| 220 | + self.send_response(200) |
| 221 | + self.send_header("Content-Type", "application/json") |
| 222 | + self.end_headers() |
| 223 | + self.wfile.write(json.dumps(response).encode()) |
| 224 | +
|
| 225 | + except jwt.InvalidTokenError as e: |
| 226 | + self.send_error(401, f"Invalid token: {str(e)}") |
| 227 | + except Exception as e: |
| 228 | + self.send_error(400, str(e)) |
| 229 | +
|
| 230 | +
|
| 231 | +if __name__ == "__main__": |
| 232 | + # In production, use a proper HTTPS server with valid certificates |
| 233 | + server = HTTPServer(("", PORT), AdapterHandler) |
| 234 | + print(f"Adapter listening on port {PORT}") |
| 235 | + server.serve_forever() |
| 236 | +``` |
| 237 | + |
| 238 | +To use this adapter: |
| 239 | + |
| 240 | +```bash |
| 241 | +# Install dependencies |
| 242 | +pip install pyjwt cryptography |
| 243 | +
|
| 244 | +# Set environment variables with your secrets |
| 245 | +export MY_API_KEY="secret-value-123" |
| 246 | +
|
| 247 | +# Run the adapter (in production, use proper HTTPS) |
| 248 | +python adapter.py |
| 249 | +``` |
| 250 | + |
| 251 | +ESC configuration: |
| 252 | + |
| 253 | +```yaml |
| 254 | +values: |
| 255 | + mySecrets: |
| 256 | + fn::open::external: |
| 257 | + url: https://my-adapter.example.com/fetch-secrets |
| 258 | + request: |
| 259 | + secretName: MY_API_KEY |
| 260 | +``` |
| 261 | + |
| 262 | +## Schema Reference |
| 263 | + |
| 264 | +### Inputs |
| 265 | + |
| 266 | +| Property | Type | Description | Required | Default | |
| 267 | +|-----------|---------|--------------------------------------------|----------|---------| |
| 268 | +| `url` | string | HTTPS URL to your adapter service | Yes | - | |
| 269 | +| `request` | object | Arbitrary JSON object sent to your adapter | No | `{}` | |
| 270 | +| `secret` | boolean | Whether to mark the response as secret | No | `true` | |
| 271 | + |
| 272 | +### Outputs |
| 273 | + |
| 274 | +| Property | Type | Description | |
| 275 | +|------------|--------|---------------------------------------------| |
| 276 | +| `response` | object | The JSON response from your adapter service | |
0 commit comments