Skip to content

cave-auth-endpoints.md

Latest commit

 

History

History
566 lines (381 loc) · 23.4 KB

cave-auth-endpoints.md

File metadata and controls

566 lines (381 loc) · 23.4 KB
Raw
doc_status living-reference
sync_policy Update with CAVE-compatible endpoint, token delivery, SCIM, and routing contract changes.
last_reviewed 2026-06-01

CAVE Authorization System: Preliminary Compatibility Guide

This document describes the HTTP API surface of the CAVE authorization system (middle_auth) and DatasetGateway's preliminary compatibility layer. CAVE support should still be treated as planned rather than certified until the implementation is tested against a real CAVE deployment and reviewed.

Architecture Overview

CAVE authorization is cleanly separated into two components:

  1. middle_auth — The Flask server providing OAuth login, token management, user/group/permission CRUD, and token validation endpoints.
  2. middle_auth_client — A Python library installed in every CAVE service, providing Flask decorators (@auth_required, @auth_requires_permission, etc.) that make HTTP callbacks to the middle_auth server on every request.

Every CAVE service delegates auth entirely through middle_auth_client. No service implements its own token validation or permission logic. The decorators call back to middle_auth, cache the result (TTL 300s by default), and gate access based on the response.

Service Discovery

Services locate middle_auth via environment variables set in Kubernetes deployment config:

  • AUTH_URL — Base URL of the auth server (e.g., cave.example.org/auth). Used by middle_auth_client decorators and by PyChunkedGraph's direct HTTP calls.
  • STICKY_AUTH_URL — URL for OAuth browser redirects (may differ for sticky session routing). Used by AnnotationFrameworkInfoService.

For a fresh deployment or planned migration, pointing CAVE services at DatasetGateway is primarily a deployment config change: update these env vars and make sure clients obtain DSG-minted tokens. Existing Bearer-token flows through middle_auth_client should continue to work, while cookie/query-token flows need the token-name migration described below.

Token Delivery

Tokens reach CAVE services via three mechanisms (checked in this order by DatasetGateway's TokenAuthentication):

  1. HTTP cookie: dsg_token
  2. Authorization header: Bearer {token}
  3. Query parameter: ?dsg_token=...

Note: The original CAVE middle_auth_client uses middle_auth_token as the cookie and query param name. DatasetGateway unifies to dsg_token. CAVE services using middle_auth_client with Bearer header auth should be unaffected by that name change; clients relying on cookie or query param tokens need a DSG login/token transition, compatibility configuration, or an updated client.

Endpoints Required for CAVE Compatibility

DatasetGateway implements preliminary versions of the small set of endpoints that CAVE services actually call. The remaining management endpoints in middle_auth are primarily used by its own admin UI and can use DatasetGateway's native UI/API design instead.

Critical Path: Token Validation and Permission Checking

Called on every authenticated request across all CAVE services via middle_auth_client decorators.

GET /api/v1/user/cache

The single most important endpoint. Every @auth_required and @auth_requires_permission decorator calls this to validate the token and retrieve the user's permissions.

Request:

  • Header: Authorization: Bearer {token}

Response (200):

{
  "id": 42,
  "parent_id": null,
  "service_account": false,
  "name": "username",
  "email": "user@example.org",
  "admin": false,
  "pi": "",
  "affiliations": [],
  "groups": ["group1", "group2"],
  "groups_admin": [],
  "permissions": {
    "fish2": 2,
    "fanc": 1
  },
  "permissions_v2": {
    "fish2": ["view", "edit"],
    "fanc": ["view"]
  },
  "permissions_v2_ignore_tos": {
    "fish2": ["view", "edit"]
  },
  "missing_tos": [],
  "datasets_admin": ["fish2"]
}

Field notes:

  • parent_id / service_account — DatasetGateway service accounts are separate ServiceAccount rows and return service_account: true, parent_id: null; User rows always return service_account: false, parent_id: null. parent_id is retained in the blob for wire compatibility only — it is now constant null for every principal (the user-type/parent-linked service account mechanism was removed)
  • permissions — legacy v1 format mapping dataset name to a numeric level (0=none, 1=view, 2=edit); the max level across all permissions for that dataset
  • permissions_v2 — permissions filtered by TOS acceptance
  • permissions_v2_ignore_tos — permissions regardless of TOS acceptance
  • missing_tos — list of {dataset_id, dataset_name, tos_id, tos_name} for datasets where the user has permissions but hasn't accepted the required TOS
  • groups_admin — groups where the user has the admin role
  • affiliations — list of affiliation names from the Affiliation model; empty for users without imported affiliations and for service accounts

Response (401): Token invalid or expired.

Caching: middle_auth_client caches responses client-side for 300 seconds (configurable via TOKEN_CACHE_TTL env var, LRU size via TOKEN_CACHE_MAXSIZE, default 1024).

Consumers: Every CAVE service with auth decorators — AnnotationEngine, MaterializationEngine, PyChunkedGraph, SkeletonService, PCGL2Cache, NeuroglancerJsonServer, AnnotationFrameworkInfoService, ProofreadingProgress, guidebook, dash_on_flask.

GET /api/v1/service/{namespace}/table/{table_id}/dataset

Maps a service table to a dataset name. Called by @auth_requires_permission when a service needs to resolve which dataset a table belongs to for permission checking.

Request:

  • Header: Authorization: Bearer {token}
  • Path params: namespace (e.g., "aligned_volume", "datastack"), table_id (string)

Response (200):

"fish2"

Consumers: Services using @auth_requires_permission with resource_namespace — MaterializationEngine (~30 endpoints), AnnotationEngine, PyChunkedGraph, SkeletonService, PCGL2Cache, AnnotationFrameworkInfoService, guidebook, dash_on_flask.

Utility Functions Called by Specific Services

GET /api/v1/user/{user_id}/permissions

Called by users_share_common_group() to check whether two users share a common group.

Request:

  • Header: Authorization: Bearer {service_token}
  • Path param: user_id (integer)

Response (200): User object with group membership.

Consumers: AnnotationEngine and MaterializationEngine only.

GET /api/v1/username?id={id1},{id2},...

Returns display names for a list of user IDs.

Request:

  • Header: Authorization: Bearer {token}
  • Query param: comma-separated user IDs

Response (200):

[
  {"id": 42, "name": "alice"},
  {"id": 43, "name": "bob"}
]

Consumers: PyChunkedGraph only (direct HTTP call in get_username_dict()).

GET /api/v1/user?id={id1},{id2},...

Returns full user info for a list of user IDs.

Request:

  • Header: Authorization: Bearer {token}
  • Query param: comma-separated user IDs

Response (200): Array of user objects.

Consumers: PyChunkedGraph only (direct HTTP call in get_userinfo_dict()).

Public Data Access

Called when checking whether unauthenticated users can access specific data.

GET /api/v1/table/{table_id}/has_public

Check if a table has any public entries.

Request: no authentication required by the current DatasetGateway view; an Authorization header may still be sent by existing clients.

Response (200): Boolean.

Cached: 300 seconds in middle_auth_client.

GET /api/v1/table/{table_id}/root/{root_id}/is_public

Check if a specific root is public.

Request: no authentication required by the current DatasetGateway view; an Authorization header may still be sent by existing clients.

Response (200): Boolean.

Cached: 300 seconds in middle_auth_client.

POST /api/v1/table/{table_id}/root_all_public

Batch check which roots are public.

Request:

  • No authentication required by the current DatasetGateway view; an Authorization header may still be sent by existing clients.
  • Header: Content-Type: application/json
  • Body: JSON array of root IDs

Response (200): list of booleans in the same order as the requested root IDs.

Cached: 300 seconds in middle_auth_client.

OAuth Flow (Browser-Facing)

Required for user login via browser.

GET/POST /api/v1/authorize

Initiates Google OAuth flow. Returns authorization URL (for programmatic clients via X-Requested-With header) or redirects browser.

Query params: redirect (return URL), tos_id (optional ToS to accept).

GET /api/v1/oauth2callback

Google OAuth callback. Exchanges code for token, creates/updates user, sets dsg_token cookie (7-day TTL), redirects to original URL.

GET/POST /api/v1/logout

Clears the dsg_token cookie and deletes the matching cookie token when present. Bearer tokens are managed through the token endpoints/UI; logout is not a service-account token revocation endpoint.

Consumers: AnnotationFrameworkInfoService redirects users to STICKY_AUTH_URL + /api/v1/logout.

Token Management (CAVEclient User-Facing)

Called by the CAVEclient Python library for programmatic token management.

POST /api/v1/create_token

Generate a new API token for the authenticated user. Always creates a new no-expiry APIKey row on every call. Use this for explicit token-management workflows (CAVEclient, middle_auth compatibility) where the caller wants a fresh token.

Request: Header: Authorization: Bearer {token}

Response (200): New token string.

GET /api/v1/long_lived_token

Return the authenticated user's stable long-lived API token. Creates the token on first call (description Default long-lived API token, expires_at = NULL) and returns the same token on every subsequent call. This is the endpoint integrated frontends should proxy to when they want to display a token for users to paste into scripts and clients (neuprint-python, clio scripts, curl).

The OAuth login token row backing the dsg_token browser cookie is never selected as the long-lived token, so calling this endpoint does not rotate the browser session.

Request: standard DSG authentication via dsg_token cookie, Bearer token, or supported query token.

Response (200):

{ "token": "<stable-token>" }

POST /api/v1/long_lived_token/rotate

Revoke the authenticated user's stable long-lived token and return a replacement. Every default long-lived row (Default long-lived API token, expires_at = NULL) for the user is deleted and a new one is created in the same transaction; the browser-session login token, explicitly created tokens, and delegated site tokens are untouched. Integrated frontends proxy this to offer a "revoke" action next to the displayed token (neuPrintHTTP: POST /token/rotate). Audited as api_token_rotated.

Services that cache token validation (neuPrintHTTP, clio-store, DVID) keep honoring the old token until their caches expire, typically 5-10 minutes.

Request: Header: Authorization: Bearer {token}, where {token} is the default long-lived token being rotated. The request must authenticate with this header; authentication by the dsg_token cookie alone is refused with 403, so a cross-site form post cannot rotate a user's token. Any other bearer (a login token, an explicitly created token) is refused with 403, so a proxy that evicts the presented token from its caches evicts exactly the revoked one. Human, non-delegated callers only.

Response (200):

{ "token": "<new-token>" }

Response (409): the token was rotated by a concurrent request (two tabs, a double click). Nothing changed; fetch the current token with GET /api/v1/long_lived_token (for example via the browser session) instead of retrying with the old one.

GET /api/v1/user/token

List all tokens for the current user.

GET /api/v1/refresh_token

Deprecated but still referenced in CAVEclient.

SCIM 2.0 Provisioning Endpoints

The scim branch of middle_auth implements SCIM 2.0 (System for Cross-domain Identity Management) endpoints for machine-to-machine provisioning of users, groups, and datasets.

Overview

SCIM 2.0 (RFC 7643 / RFC 7644) provides a standardized REST API for identity provisioning. These endpoints enable external identity providers (e.g., Okta, Azure AD) to automatically manage users, groups, and dataset assignments without using the admin UI.

Base URL

/{URL_PREFIX}/scim/v2

Default: /auth/scim/v2

Authentication

All SCIM endpoints require a Bearer token for a non-expired APIKey belonging to an active DatasetGateway User with admin=True, enforced by SCIMAuthentication:

Authorization: Bearer {admin_token}

Discovery Endpoints

GET /auth/scim/v2/ServiceProviderConfig

Returns the SCIM service provider configuration, including supported features (patch, bulk, filter, sort, changePassword, etag).

GET /auth/scim/v2/ResourceTypes

Returns the list of supported resource types: User, Group, and Dataset (custom).

GET /auth/scim/v2/Schemas

Returns full JSON schemas for all supported resource types.

User CRUD

Method Path Description
GET /auth/scim/v2/Users List users (with filtering and pagination)
GET /auth/scim/v2/Users/{scim_id} Get a single user
POST /auth/scim/v2/Users Create a new user
PUT /auth/scim/v2/Users/{scim_id} Replace a user (full update)
PATCH /auth/scim/v2/Users/{scim_id} Partially update a user
DELETE /auth/scim/v2/Users/{scim_id} Hard-delete a user

Schema URNs:

  • urn:ietf:params:scim:schemas:core:2.0:User
  • urn:ietf:params:scim:schemas:extension:neuroglancer:2.0:User (custom extension with admin, pi, gdprConsent, serviceAccount fields)

Key field mappings:

  • userName → email
  • displayName / name.formatted → name
  • active → is_active for create/replace/patch soft-disable flows; DELETE /Users/{scim_id} hard-deletes the user
  • externalId → external_id
  • id → scim_id (UUID5 deterministic from internal ID)

Group CRUD

Method Path Description
GET /auth/scim/v2/Groups List groups (with filtering and pagination)
GET /auth/scim/v2/Groups/{scim_id} Get a single group (includes members)
POST /auth/scim/v2/Groups Create a new group
PUT /auth/scim/v2/Groups/{scim_id} Replace a group (full update)
PATCH /auth/scim/v2/Groups/{scim_id} Partially update a group (add/remove members)
DELETE /auth/scim/v2/Groups/{scim_id} Delete a group

Schema URN: urn:ietf:params:scim:schemas:core:2.0:Group

Key field mappings:

  • displayName → name
  • members → UserGroup M:M relationship (each member has value = user SCIM ID and display = user name)

Dataset CRUD (Custom Resource)

Method Path Description
GET /auth/scim/v2/Datasets List datasets (with filtering and pagination)
GET /auth/scim/v2/Datasets/{scim_id} Get a single dataset
POST /auth/scim/v2/Datasets Create a new dataset
PUT /auth/scim/v2/Datasets/{scim_id} Replace a dataset (full update)
PATCH /auth/scim/v2/Datasets/{scim_id} Partially update a dataset
DELETE /auth/scim/v2/Datasets/{scim_id} Delete a dataset

Schema URN: urn:ietf:params:scim:schemas:neuroglancer:2.0:Dataset (custom)

Key field mappings:

  • name → name (dataset slug)
  • tosId → tos_id (FK to TOS document)
  • serviceTables → associated ServiceTable records (service name + table name pairs)

ID Mapping

SCIM IDs are deterministic UUID5 values generated from internal integer IDs:

uuid5(SCIM_NAMESPACE, f"{resource_type}:{internal_id}")

Where SCIM_NAMESPACE = uuid.UUID("6ba7b810-9dad-11d1-80b4-00c04fd430c8").

Each resource also stores an optional externalId field for mapping to the identity provider's internal ID. Current detail routes look up resources by the SCIM id in the path; they do not fall back to externalId.

Filtering

Supports RFC 7644 filter expressions on list endpoints via the filter query parameter:

GET /auth/scim/v2/Users?filter=userName eq "user@example.org"
GET /auth/scim/v2/Groups?filter=displayName co "admin"

Supported operators: eq, ne, co (contains), sw (starts with), ew (ends with), pr (present), gt, ge, lt, le. Logical operators and, or, and not are supported.

Pagination

SCIM uses 1-based pagination with startIndex and count parameters:

GET /auth/scim/v2/Users?startIndex=1&count=50
  • startIndex defaults to 1 (1-based index)
  • count defaults to 100, max 1000
  • count=0 returns the standard ListResponse envelope with totalResults, itemsPerPage: 0, and an empty Resources array

Response format:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
  "totalResults": 150,
  "itemsPerPage": 50,
  "startIndex": 1,
  "Resources": [...]
}

Error Response Format

SCIM errors use application/scim+json content type:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:Error"],
  "status": "404",
  "scimType": "invalidValue",
  "detail": "Resource not found"
}

New Model Fields

The SCIM implementation adds the following fields to existing models:

Model Field Type Description
User scim_id String(36), unique, indexed UUID5 SCIM identifier
User external_id String(255), unique, indexed External system identifier
Group scim_id String(36), unique, indexed UUID5 SCIM identifier
Group external_id String(255), unique, indexed External system identifier
Dataset scim_id String(36), unique, indexed UUID5 SCIM identifier
Dataset external_id String(255), unique, indexed External system identifier

Endpoints NOT Required for CAVE Service Compatibility

The following endpoint groups are only called by middle_auth's own admin UI. A replacement can implement equivalent functionality with any API design:

  • Group CRUD — POST/GET/PUT/DELETE /api/v1/group/...
  • Dataset CRUD — POST/GET/PUT/DELETE /api/v1/dataset/...
  • Permission CRUD — POST/GET/PUT /api/v1/permission/...
  • Service account management — POST/GET/PUT/DELETE /api/v1/service_account/...
  • Terms of Service management — POST/GET/PUT /api/v1/tos/...
  • Service-table-dataset mapping management — POST/DELETE /api/v1/service/{service}/table/{table}/dataset/{dataset}
  • Redis debugging — GET /api/v1/redis/...
  • User admin operations — POST/PUT/DELETE /api/v1/user/... (the admin CRUD, not /user/cache which is critical)

Decorator Reference

These are the middle_auth_client decorators used by CAVE services. A replacement that keeps the middle_auth_client library (or a fork) needs to satisfy the HTTP calls these decorators make.

Decorator HTTP Calls Used By
@auth_required GET /api/v1/user/cache All 10 services
@auth_requires_permission(perm, table_arg, resource_namespace) GET /api/v1/user/cache + GET /api/v1/service/{ns}/table/{id}/dataset MaterializationEngine (30+), PyChunkedGraph, SkeletonService, AnnotationEngine, PCGL2Cache, AnnotationFrameworkInfoService, guidebook, dash_on_flask
@auth_requires_admin GET /api/v1/user/cache (checks admin field) MaterializationEngine, AnnotationEngine, AnnotationFrameworkInfoService, NeuroglancerJsonServer
@auth_requires_dataset_admin GET /api/v1/user/cache + GET /api/v1/service/{ns}/table/{id}/dataset (checks datasets_admin) MaterializationEngine (2 endpoints)
@auth_requires_group(group) GET /api/v1/user/cache (checks groups field) Not currently used by any service

Replacement Strategies

Strategy A: Replace the server, keep middle_auth_client

Implement the endpoint contract above and keep the existing middle_auth_client decorators in CAVE services. For a fresh deployment or planned migration where users and automation obtain DSG-minted Bearer tokens, this should be mostly a deployment configuration change (AUTH_URL / STICKY_AUTH_URL). It is not yet certified as drop-in support: the implementation still needs real CAVE deployment testing, and cookie/query-token flows that depend on middle_auth_token need a DSG token transition or compatibility layer. The SCIM 2.0 endpoints add provisioning APIs for machine-to-machine management.

Strategy B: Replace both server and client

If you want to change the auth model (e.g., local JWT validation instead of per-request HTTP callback to /user/cache), fork middle_auth_client to do local token validation. This eliminates the HTTP round-trip on every request but requires updating the dependency in every CAVE service.

Multi-Dataset / Multi-Platform Routing

A central auth service handling multiple CAVE deployments (and non-CAVE services) needs to distinguish which dataset a request belongs to without endpoint conflicts.

Recommended: Path prefix routing

Add the dataset and service type as a path prefix. The original /api/v1/... paths remain unchanged beneath it:

basedomain.org/fish2/cave/api/v1/user/cache
basedomain.org/fanc/cave/api/v1/user/cache
basedomain.org/other-service/custom/api/v1/...

Each CAVE deployment sets its AUTH_URL to include the prefix:

AUTH_URL=basedomain.org/fish2/cave

The auth service parses the prefix to determine dataset context. Non-CAVE services get their own prefixes with no conflict.

Advantages:

  • Single domain, single TLS certificate
  • Standard reverse proxy routing
  • AUTH_URL already supports path prefixes — no code changes needed
  • Fork of middle_auth_client not required if proxy strips the prefix before forwarding

Alternative: Reverse proxy with prefix stripping

Keep paths identical (/api/v1/...) and use separate AUTH_URL values per deployment. A reverse proxy routes based on prefix and injects the dataset context as a header:

AUTH_URL=basedomain.org/fish2  → proxy strips /fish2, adds X-Dataset: fish2 header
AUTH_URL=basedomain.org/fanc   → proxy strips /fanc, adds X-Dataset: fanc header

The auth service receives unmodified /api/v1/... paths with dataset context in a header. No CAVE code changes or middle_auth_client fork required.

Not recommended: Subdomain encoding

Encoding the dataset in the subdomain (fish2-cave.basedomain.org) works but requires wildcard DNS records, wildcard TLS certificates, and subdomain parsing logic. The path-based approaches are simpler operationally.

Source Code References

  • Auth server: middle_auth
  • Client library: middle_auth_client
  • Token validation and caching: middle_auth_client/decorators.py — user_cache_http(), @auth_required
  • Permission checking: middle_auth_client/decorators.py — @auth_requires_permission, dataset_from_table_id()
  • PyChunkedGraph direct calls: pychunkedgraph/app/app_utils.py — get_username_dict(), get_userinfo_dict()
  • CAVEclient endpoint definitions: caveclient/endpoints.py — auth_endpoints_v1
  • Environment variable configuration: Set via Kubernetes deployment config (see CAVEdeployment)