Add connectors for third-party integrations

[heeki]

Add connectors for third-party integrations

Summary

Add a connectors framework that allows administrators to configure third-party (3P) service integrations and control which users can access them, while allowing end users to enable those connectors through a standard OAuth2 authorization code flow. User tokens are cached to eliminate repeated consent prompts, and users can toggle connectors off with an option to clear cached tokens.

Context

The platform already supports OAuth2-protected external resources for MCP servers (backend/app/models/mcp.py) and A2A agents (backend/app/models/a2a.py). Both follow the same pattern: an administrator configures the endpoint and OAuth2 credentials, an access control table (McpServerAccess, A2aAgentAccess) maps personas to the resource, and a frontend admin page (McpServersPage.tsx, A2aAgentsPage.tsx) exposes CRUD and an access control tab (McpAccessControl.tsx, A2aAccessControl.tsx).

The connectors feature extends this pattern with a user-initiated OAuth2 authorization code flow. Rather than the backend authenticating on behalf of all users with a shared client secret, each end user individually authorizes the connector via their own 3P identity provider login. The resulting per-user tokens (access token, refresh token, and optionally ID token) are cached in the backend so that subsequent invocations do not require re-consent.

Key differences from MCP/A2A:

• Administrator configures the connector definition (OAuth2 app registration, scopes, redirect URI) and access rules.
• End user enables the connector, which triggers a redirect to the 3P login/consent page.
• Per-user tokens are stored in a new user_connector_tokens table, not shared credentials.
• Users can disable a connector without clearing tokens (default) or choose to purge cached tokens.

Requirements

R1: Administrator Connector Management

R1.1: Connector Model and Database Table

Create a new Connector ORM model in backend/app/models/connector.py that stores connector definitions managed by administrators.

Fields:

• id — auto-increment primary key
• name — display name (e.g., "Google Drive", "Slack")
• description — human-readable description of what value the connector provides to users
• logo_url — URL of the connector's logo/icon for display in the UI (nullable)
• auth_type — always "oauth2" for connectors; validated at the model layer
• oauth2_well_known_url — OpenID Connect discovery document URL or equivalent
• oauth2_authorization_endpoint — authorization endpoint (derived from well-known or manually specified)
• oauth2_token_endpoint — token endpoint (derived from well-known or manually specified)
• oauth2_client_id — client ID registered with the 3P identity provider
• oauth2_client_secret — client secret (never returned in API responses)
• oauth2_scopes — space-separated list of requested scopes
• oauth2_redirect_uri — the redirect URI registered with the 3P identity provider (must match)
• status — active or inactive; inactive connectors are hidden from end users
• created_at, updated_at

Relationships:

• access_rules → ConnectorAccess (one-to-many, cascade delete)
• user_tokens → UserConnectorToken (one-to-many, cascade delete)

R1.2: Connector Access Control Model

Create a ConnectorAccess ORM model that maps personas (groups) to connectors, following the pattern in McpServerAccess (backend/app/models/mcp.py:81-114).

Fields:

• id, connector_id (FK → connectors.id), persona_id, created_at, updated_at

A user can see and enable a connector only if their persona (group) has an access rule for it.

R1.3: User Connector Token Model

Create a UserConnectorToken ORM model in backend/app/models/connector.py to cache per-user OAuth2 tokens.

Fields:

• id, connector_id (FK → connectors.id), user_sub (Cognito subject)
• enabled — boolean; false means the user has toggled it off
• access_token — encrypted at rest (see R1.9)
• refresh_token — encrypted at rest
• id_token — encrypted at rest (nullable)
• token_expires_at — datetime of access token expiry
• created_at, updated_at

Unique constraint: (connector_id, user_sub).

R1.4: Admin CRUD API for Connectors

Create a new router backend/app/routers/connectors.py (prefix /api/connectors, tags ["connectors"]), following the pattern in backend/app/routers/mcp.py.

Admin endpoints (require connectors:write scope):

• POST /api/connectors — create connector
• GET /api/connectors — list all connectors (require connectors:read)
• GET /api/connectors/{id} — get connector by ID (require connectors:read)
• PUT /api/connectors/{id} — update connector
• DELETE /api/connectors/{id} — delete connector and cascade-delete all access rules and user tokens

The oauth2_client_secret field must never be included in response payloads. Return has_oauth2_secret: bool instead, matching the pattern in McpServer.to_dict() (backend/app/models/mcp.py:41).

R1.5: Admin Access Control API

Add access rule endpoints to the connectors router, following the pattern in backend/app/routers/mcp.py:309-351.

Endpoints (require connectors:write for mutations, connectors:read for reads):

• GET /api/connectors/{id}/access — list access rules for this connector
• PUT /api/connectors/{id}/access — replace all access rules (list of { persona_id })

R1.6: Admin Frontend — Connectors Management Page

Create a new page frontend/src/pages/ConnectorsPage.tsx for administrators, following the structure of frontend/src/pages/McpServersPage.tsx.

Features:

• Card and table view modes (consistent with other resource pages)
• Add/edit/delete connectors via a ConnectorForm component (see R1.7)
• Detail panel with two tabs: Settings (edit form) and Access (access control component, see R1.8)
• Display connector logo, name, description, OAuth2 configuration summary, and status badge on each card/row

R1.7: Connector Form Component

Create frontend/src/components/ConnectorForm.tsx for creating and editing connectors.

Fields:

• Name, description, logo URL
• OAuth2 well-known URL (with a "Discover" button that fetches the discovery document and auto-fills authorization and token endpoints)
• Authorization endpoint, token endpoint (auto-filled or manually overridden)
• Client ID, client secret (password field), scopes, redirect URI
• Status toggle (active/inactive)

R1.8: Connector Access Control Component

Create frontend/src/components/ConnectorAccessControl.tsx, following the pattern in frontend/src/components/McpAccessControl.tsx.

Behavior:

• Lists all registered agents/personas; each can be toggled on or off to grant or deny access to this connector.
• Unlike McpAccessControl, there is no tool-level granularity — access is binary (enabled or disabled per persona).
• Save button replaces all access rules atomically via PUT /api/connectors/{id}/access.

R1.9: Token Encryption at Rest

User tokens stored in UserConnectorToken must be encrypted before persisting. Use a symmetric encryption scheme (e.g., AES-256-GCM via the cryptography library's Fernet or AESGCM) with an encryption key sourced from an environment variable (LOOM_TOKEN_ENCRYPTION_KEY).

• Provide a helper module backend/app/services/token_crypto.py with encrypt(plaintext: str) -> str and decrypt(ciphertext: str) -> str.
• Apply encryption in UserConnectorToken property setters for access_token, refresh_token, and id_token.
• Document the required environment variable in etc/environment.sh.

---

R2: User Connectors Section

R2.1: User-Facing Connectors API

Add user-facing endpoints to backend/app/routers/connectors.py.

Endpoints (require authenticated user, no special admin scope):

GET /api/connectors/me — returns the list of connectors accessible to the calling user (based on their persona/group membership from Cognito) with each connector's enabled state and token status for that user. Never returns token values.

Response shape per connector:

{
  "id": 1,
  "name": "Google Drive",
  "description": "...",
  "logo_url": "...",
  "oauth2_scopes": "...",
  "status": "active",
  "user_enabled": true,
  "token_cached": true,
  "token_expires_at": "2026-04-01T00:00:00Z"
}

GET /api/connectors/{id}/authorize — initiates the OAuth2 authorization code flow for the calling user. Generates a state parameter (CSRF token, stored server-side in a short-lived cache or signed JWT), builds the authorization URL with response_type=code, client_id, redirect_uri, scope, and state, and returns it to the frontend.

GET /api/connectors/callback — OAuth2 redirect URI handler. Validates state, exchanges code for tokens via the connector's token endpoint, encrypts and stores tokens in UserConnectorToken, marks enabled=true. Redirects the user's browser back to the frontend connectors page with a success indicator in the query string.

PUT /api/connectors/{id}/disable — marks enabled=false on the user's UserConnectorToken record. Accepts a request body { "clear_tokens": bool }:

• If clear_tokens=false (default), set enabled=false but retain cached tokens.
• If clear_tokens=true, delete the UserConnectorToken row entirely.

POST /api/connectors/{id}/refresh — exchanges the cached refresh token for a new access token, updates access_token and token_expires_at. Returns updated token metadata (not the token itself). Used by the frontend to silently refresh before an access token expires.

R2.2: Frontend Connectors Page for End Users

Create frontend/src/pages/ConnectorsPage.tsx (or a separate user-facing route, e.g., /connectors) that lists the connectors accessible to the current user.

Layout per connector card:

• Connector logo (if logo_url is set) or a generic plug/link icon from lucide-react as fallback
• Connector name and description — the description must make it clear what value the connector provides (e.g., "Connect Google Drive to let the agent search, read, and summarize your documents.")
• Status badge: Connected (green) when user_enabled=true and token_cached=true, Token expired (yellow) when the token is cached but expired, Not connected (muted) otherwise
• Toggle switch to enable or disable the connector

Enable flow:

1. User toggles the switch on.
2. Frontend calls GET /api/connectors/{id}/authorize to receive the authorization URL.
3. Frontend redirects (or opens in the same tab) to the authorization URL at the 3P identity provider.
4. After the user logs in and consents, the 3P provider redirects to the backend callback URL.
5. Backend exchanges the code, stores tokens, and redirects the browser to the frontend connectors page.
6. Frontend detects the success indicator in the query string and refreshes the connector list, showing the connector as Connected.

Disable flow:

1. User toggles the switch off.
2. A confirmation dialog appears: "Disconnect [Connector Name]?" with two options:
   • Disconnect (default) — calls PUT /api/connectors/{id}/disable with clear_tokens=false.
   • Disconnect and clear tokens — calls PUT /api/connectors/{id}/disable with clear_tokens=true.
3. Connector card updates to Not connected status.

R2.3: OAuth2 Callback Route in the Frontend

Register the OAuth2 callback redirect URI in the frontend router (frontend/src/App.tsx) if the callback is handled by the backend directly (redirect to frontend after token exchange). The callback page should:

• Read the success/error query parameter from the URL.
• Show a loading indicator while the connector list refreshes.
• Display a success toast ("Connected to [Connector Name]") or an error message.
• Clear the query parameter from the URL after processing (use history.replaceState or the router's navigation API).

R2.4: Token Refresh Handling

Before a connector's access token expires, the frontend should silently refresh it.

Desired behavior:

• When rendering the connectors list, if token_expires_at is within 5 minutes of the current time and user_enabled=true, call POST /api/connectors/{id}/refresh in the background.
• If the refresh fails (e.g., refresh token revoked), update the connector status to Token expired and prompt the user to re-authorize by toggling the connector off and on.

---

R3: Navigation and Scopes

R3.1: Register Connectors Router in the Backend

Register the new connectors router in backend/app/main.py, following the pattern for existing routers.

R3.2: Add Connectors Scopes

Add connectors:read and connectors:write scopes to the scope definitions used by backend/app/dependencies/auth.py and wherever other scopes (e.g., mcp:read, mcp:write) are defined or documented.

R3.3: Add Connectors Navigation Entry

Add a navigation entry for the admin Connectors management page in the sidebar/nav consistent with the MCP Servers and A2A Agents nav entries. Locate the nav definition by searching for McpServersPage or A2aAgentsPage in frontend/src/App.tsx.

Add a user-facing Connectors page entry in the user navigation section, visible to users whose persona has access to at least one active connector.

R3.4: Backend Tests

Add unit tests in backend/tests/test_connectors.py following the pattern in backend/tests/test_a2a.py, covering:

• CRUD operations for connector definitions
• Access rule creation and retrieval
• The authorize endpoint returns a valid authorization URL
• The disable endpoint with clear_tokens=false retains the token row
• The disable endpoint with clear_tokens=true deletes the token row
• Unauthenticated requests are rejected

Files to Create

File	Description
backend/app/models/connector.py	Connector, ConnectorAccess, UserConnectorToken ORM models
backend/app/routers/connectors.py	Admin and user-facing API endpoints
backend/app/services/token_crypto.py	Encrypt/decrypt helpers for token storage
backend/tests/test_connectors.py	Unit tests
frontend/src/pages/ConnectorsPage.tsx	Admin management page
frontend/src/pages/UserConnectorsPage.tsx	End-user connectors toggle page
frontend/src/components/ConnectorForm.tsx	Create/edit connector form
frontend/src/components/ConnectorAccessControl.tsx	Per-persona access rules panel
frontend/src/api/connectors.ts	Frontend API client functions
frontend/src/hooks/useConnectors.ts	React hook for connector state

Files to Modify

File	Changes
backend/app/main.py	Register connectors.router
backend/app/models/__init__.py	Import and expose new connector models
backend/app/db.py	Ensure new tables are created on startup
frontend/src/App.tsx	Add connectors routes (admin and user-facing) and nav entries
frontend/src/api/types.ts	Add Connector, ConnectorAccess, UserConnector TypeScript types
etc/environment.sh	Document LOOM_TOKEN_ENCRYPTION_KEY environment variable

Acceptance Criteria

• R1.1: Connector model exists with all required fields; oauth2_client_secret is never returned in API responses
• R1.2: ConnectorAccess model exists; access rules are persona-scoped
• R1.3: UserConnectorToken model exists with unique constraint on (connector_id, user_sub)
• R1.4: Admin can create, read, update, and delete connectors via the API
• R1.5: Admin can set per-persona access rules via PUT /api/connectors/{id}/access
• R1.6: Admin connectors management page exists with card and table views
• R1.7: Connector form validates required OAuth2 fields; "Discover" button auto-fills endpoints from well-known URL
• R1.8: Connector access control panel shows all personas with enable/disable toggle; save persists atomically
• R1.9: Tokens are encrypted at rest using LOOM_TOKEN_ENCRYPTION_KEY
• R2.1: GET /api/connectors/me returns only connectors accessible to the calling user's persona
• R2.1: GET /api/connectors/{id}/authorize returns a valid authorization URL with state parameter
• R2.1: GET /api/connectors/callback exchanges code for tokens and redirects to frontend
• R2.1: PUT /api/connectors/{id}/disable with clear_tokens=false retains token row; with clear_tokens=true deletes it
• R2.1: POST /api/connectors/{id}/refresh updates access token using refresh token
• R2.2: User connectors page displays logo or fallback icon, name, description, and status badge
• R2.2: Enabling a connector redirects the user to the 3P login page
• R2.2: After successful OAuth2 flow, connector shows as Connected without re-consent
• R2.2: Disabling shows a confirmation dialog with both "Disconnect" and "Disconnect and clear tokens" options
• R2.3: OAuth2 callback route in the frontend handles success and error query parameters and clears them from the URL
• R2.4: Access tokens within 5 minutes of expiry are silently refreshed; expired/revoked tokens surface a re-authorize prompt
• R3.1: Connectors router registered in backend/app/main.py
• R3.2: connectors:read and connectors:write scopes defined and enforced
• R3.3: Admin and user nav entries present for connectors pages
• R3.4: Backend unit tests pass for all CRUD, access rule, and OAuth2 flow scenarios
• No regressions to existing MCP, A2A, or credential provider functionality