Merge pull request #43 from mitodl/jkachel/6598-6538-add-auth-infrastructure

Add auth infrastructure, based on APISIX/Keycloak

Author: jkachel

.pre-commit-config.yaml

@@ -60,6 +60,10 @@ repos:
           - yarn.lock
           - --exclude-files
           - ".*/generated/"
+          - --exclude-files
+          - "config/keycloak/tls/*"
+          - --exclude-files
+          - "config/keycloak/realms/default-realm.json"
         additional_dependencies: ["gibberish-detector"]
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: "v0.7.2"

.secrets.baseline

@@ -100,7 +100,8 @@
         "test_.*.py",
         "poetry.lock",
         "yarn.lock",
-        ".*/generated/"
+        ".*/generated/",
+        "config/keycloak/tls/*"
       ]
     }
   ],

README-keycloak.md

@@ -0,0 +1,58 @@
+# Keycloak Integration
+
+The Compose file includes a Keycloak instance that you can use for authentication instead of spinning up a separate one or using one of the deployed instances. (It's enabled by default but running it won't prevent you from using a separate instance.)
+
+## Default Settings
+
+There are some defaults that are part of this.
+
+_SSL Certificate_: There's a self-signed cert that's in `config/keycloak/tls` - if you'd rather set up your own (or you have a real cert or something to use), you can drop the PEM files in there. See the README there for info.
+
+_Realm_: There's a `default-realm.json` in `config/keycloak` that will get loaded by Keycloak when it starts up, and will set up a realm for you with some users and a client so you don't have to set it up yourself. The realm it creates is called `ol-local`.
+
+The users it sets up are:
+
+| User                | Password  |
+| ------------------- | --------- |
+| `[email protected]` | `student` |
+| `[email protected]`    | `prof`    |
+| `[email protected]`   | `admin`   |
+
+The client it sets up is called `apisix`. You can change the passwords and get the secret in the admin.
+
+## Making it Work
+
+If you don't have a Keycloak instance running locally already, you can use the pack-in one. It starts with the rest of the services and is configured to be at `http://kc.ol.local:8006` and `https://kc.ol.local:8007` by default (but you can change this in the `env` files).
+
+Some setup is required to use the pack-in instance:
+
+1. Set required keycloak environment values in your `.env` file:
+   - Set a keystore password via `KEYCLOAK_SVC_KEYSTORE_PASSWORD`. This is required, but the password need not be anything special.
+   - Set `KEYCLOAK_CLIENT_SECRET`; ask another developer for the relevant value.
+2. Optionally add `KEYCLOAK_SVC_HOSTNAME`, `KEYCLOAK_SVC_ADMIN`, and `KEYCLOAK_SVC_ADMIN_PASSWORD` to your `.env` file.
+   1. `KEYCLOAK_SVC_HOSTNAME` is the hostname you want to use for the instance - the default is `kc.ol.local`.
+   2. `KEYCLOAK_SVC_ADMIN` is the admin username. The default is `admin`.
+   3. `KEYCLOAK_SVC_ADMIN_PASSWORD` is the admin password. The default is `admin`.
+3. Re-start the stack.
+
+The Keycloak container should start and stay running. Once it does, you should be able to log in at `https://kc.ol.local:8007` with username and password `admin` (or the values you supplied).
+
+If you'd rather use a separate Keycloak instance, ensure these settings are present in the appropriate `env` file (best is probably `backend.local.env`):
+
+- `KEYCLOAK_REALM`
+
+  Sets the realm used by APISIX for Keycloak authentication. Defaults to `ol-local`.
+
+- `KEYCLOAK_DISCOVERY_URL`
+
+  Sets the discovery URL for the Keycloak OIDC service. (In Keycloak admin, navigate to the realm you're using, then go to Realm Settings under Configure, and the link is under OpenID Endpoint Configuration.) This defaults to a valid value for the pack-in Keycloak instance.
+
+- `KEYCLOAK_CLIENT_ID`
+
+  The client ID for the OIDC client for APISIX. Defaults to `apisix`.
+
+- `KEYCLOAK_CLIENT_SECRET`
+
+  The client secret for the OIDC client. No default - you will need to get this from the Keycloak admin, even if you're using the pack-in Keycloak instance.
+
+> If you're using a Keycloak instance also hosted within a Docker container on the same machine you're running the AI chatbots, you'll need to make sure it can be seen from within the `apigateway` container. This will _require_ some work on your part - generally, stuff within Composer environments can't see things outside of their own environment. There's an example of this in the `docker-compose.services.yml` file if your Keycloak instance uses a Compose environment, as we use it so that the Keycloak OIDC URLs all match externally and internally.

ai_chatbots/consumers.py

@@ -55,7 +55,7 @@ async def prepare_response(self, serializer) -> ChatRequestSerializer:
         )
 
         if user and user.username and user.username != "AnonymousUser":
-            self.user_id = user.username
+            self.user_id = user.global_id.replace("-", "_")
         elif session:
             if not session.session_key:
                 session.save()

ai_chatbots/consumers_test.py

@@ -3,6 +3,7 @@
 import json
 from random import randint
 from unittest.mock import AsyncMock
+from uuid import uuid4
 
 import pytest
 
@@ -18,7 +19,8 @@
 def agent_user():
     """Return a user for the agent."""
     return UserFactory.build(
-        username=f"test_user_{randint(1, 1000)}"  # noqa: S311
+        username=f"test_user_{randint(1, 1000)}",  # noqa: S311
+        global_id=f"test_user_{uuid4()!s}",
     )
 
 
@@ -166,7 +168,7 @@ async def test_syllabus_create_chatbot(
     mock_http_consumer_send.send_headers.assert_called_once()
     chatbot = syllabus_consumer.create_chatbot(serializer)
     assert isinstance(chatbot, SyllabusBot)
-    assert chatbot.user_id == agent_user.username
+    assert chatbot.user_id == agent_user.global_id.replace("-", "_")
     assert chatbot.temperature == 0.7
     assert chatbot.instructions == "Answer this question as best you can"
     assert chatbot.model == "gpt-3.5-turbo"

ai_chatbots/routing.py

@@ -1,6 +1,7 @@
 from django.urls import re_path
 
 from ai_chatbots import consumers
+from users.consumers import UserMetaHttpConsumer
 
 http_patterns = [
     re_path(
@@ -13,4 +14,11 @@
         consumers.SyllabusBotHttpConsumer.as_asgi(),
         name="syllabus_agent_sse",
     ),
+    # This gets two routes - user_meta doesn't require auth (in the APISIX settings)
+    # and login does.
+    re_path(
+        r"^http/(user_meta|login)/$",
+        UserMetaHttpConsumer.as_asgi(),
+        name="user_meta",
+    ),
 ]

config/apisix/apisix.yaml

@@ -0,0 +1,111 @@
+upstreams:
+  - id: 1
+    nodes:
+      "nginx:${{NGINX_PORT}}": 1
+    type: roundrobin
+
+routes:
+  - id: 1
+    name: "websocket"
+    desc: "Special handling for websocket URLs."
+    priority: 1
+    upstream_id: 1
+    enable_websocket: true
+    plugins:
+      openid-connect:
+        client_id: ${{KEYCLOAK_CLIENT_ID}}
+        client_secret: ${{KEYCLOAK_CLIENT_SECRET}}
+        discovery: ${{KEYCLOAK_DISCOVERY_URL}}
+        realm: ${{KEYCLOAK_REALM}}
+        scope: ${{KEYCLOAK_SCOPES}}
+        bearer_only: false
+        introspection_endpoint_auth_method: "client_secret_post"
+        ssl_verify: false
+        session:
+          secret: ${{APISIX_SESSION_SECRET_KEY}}
+        logout_path: "/logout"
+        post_logout_redirect_uri: ${{APISIX_LOGOUT_URL}}
+        unauth_action: "pass"
+      cors:
+        allow_origins: "**"
+        allow_methods: "**"
+        allow_headers: "**"
+        allow_credential: true
+      response-rewrite:
+        headers:
+          set:
+            Referrer-Policy: "origin"
+    uris:
+      - "/ws/*"
+  - id: 2
+    name: "passauth"
+    desc: "Wildcard route that can use auth but doesn't require it."
+    priority: 0
+    upstream_id: 1
+    plugins:
+      openid-connect:
+        client_id: ${{KEYCLOAK_CLIENT_ID}}
+        client_secret: ${{KEYCLOAK_CLIENT_SECRET}}
+        discovery: ${{KEYCLOAK_DISCOVERY_URL}}
+        realm: ${{KEYCLOAK_REALM}}
+        scope: ${{KEYCLOAK_SCOPES}}
+        bearer_only: false
+        introspection_endpoint_auth_method: "client_secret_post"
+        ssl_verify: false
+        session:
+          secret: ${{APISIX_SESSION_SECRET_KEY}}
+        logout_path: "/logout"
+        post_logout_redirect_uri: ${{APISIX_LOGOUT_URL}}
+        unauth_action: "pass"
+      cors:
+        allow_origins: "**"
+        allow_methods: "**"
+        allow_headers: "**"
+        allow_credential: true
+      response-rewrite:
+        headers:
+          set:
+            Referrer-Policy: "origin"
+    uri: "*"
+  - id: 3
+    name: "logout-redirect"
+    desc: "Strip trailing slash from logout redirect."
+    priority: 10
+    upstream_id: 1
+    uri: "/logout/*"
+    plugins:
+      redirect:
+        uri: "/logout"
+  - id: 4
+    name: "reqauth"
+    desc: "Routes that require authentication."
+    priority: 10
+    upstream_id: 1
+    plugins:
+      openid-connect:
+        client_id: ${{KEYCLOAK_CLIENT_ID}}
+        client_secret: ${{KEYCLOAK_CLIENT_SECRET}}
+        discovery: ${{KEYCLOAK_DISCOVERY_URL}}
+        realm: ${{KEYCLOAK_REALM}}
+        scope: ${{KEYCLOAK_SCOPES}}
+        bearer_only: false
+        introspection_endpoint_auth_method: "client_secret_post"
+        ssl_verify: false
+        session:
+          secret: ${{APISIX_SESSION_SECRET_KEY}}
+        logout_path: "/logout"
+        post_logout_redirect_uri: ${{APISIX_LOGOUT_URL}}
+        unauth_action: "auth"
+      cors:
+        allow_origins: "**"
+        allow_methods: "**"
+        allow_headers: "**"
+        allow_credential: true
+      response-rewrite:
+        headers:
+          set:
+            Referrer-Policy: "origin"
+    uris:
+      - "/admin/login/*"
+      - "/http/login/"
+#END

config/apisix/config.yaml

@@ -0,0 +1,11 @@
+apisix:
+  enable_admin: false
+  enable_dev_mode: false
+  node_listen:
+    - port: ${{APISIX_PORT}}
+
+deployment:
+  role: data_plane
+  role_data_plane:
+    config_provider: yaml
+#END

config/apisix/debug.yaml

@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+basic:
+  enable: true # Enable the basic debug mode.
+http_filter:
+  enable: false # Enable HTTP filter to dynamically apply advanced debug settings.
+  enable_header_name: X-APISIX-Dynamic-Debug # If the header is present in a request, apply the advanced debug settings.
+hook_conf:
+  enable: false # Enable hook debug trace to log the target module function's input arguments or returned values.
+  name: hook_phase # Name of module and function list.
+  log_level: warn # Severity level for input arguments and returned values in the error log.
+  is_print_input_args: true # Print the input arguments.
+  is_print_return_value: true # Print the return value.
+
+hook_phase: # Name of module and function list.
+  apisix: # Required module name.
+    - http_access_phase # Required function names.
+    - http_header_filter_phase
+    - http_body_filter_phase
+    - http_log_phase
+#END