authservice hanging using an Azure AD application

[dippynark]

I have deployed authservice as its own Deployment in its own Namespace and am trying to configure authentication to a service using an Azure AD application. The setup is working except that the POST to login.microsoftonline.com (I believe to retrieve the ID token) hangs for about 2 minutes when logging in (2m 8s with a stopwatch). Potentially this is related to this issue but I wasn't sure.

Potentially this is due to configuration on the Azure AD side, however I am using the same application to authenticate to other separate services using OIDC (e.g. Vault) and everything works as expected, so I believe there is something that can be done on the authservice side to fix this.

I have attached logs for authservice below as well as its sidecar and the sidecar of the service I want to authenticate to (Kiali) as described here.

I am using the following resources to enforce authn/z by Istio:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: kiali
  namespace: kiali
spec:
  selector:
    matchLabels:
      app: kiali
  jwtRules:
  - issuer: https://login.microsoftonline.com/<APPLICATION_ID>/v2.0
    jwksUri: https://login.microsoftonline.com/<APPLICATION_ID>/discovery/v2.0/keys
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: kiali
  namespace: kiali
spec:
  selector:
    matchLabels:
      app: kiali
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]

As well as the following EnvoyFilters to allow authservice to intervene:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authservice
  namespace: kiali
spec:
  workloadSelector:
    labels:
      app: kiali
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.jwt_authn"
    patch:
      operation: INSERT_BEFORE
      value:
       name: envoy.ext_authz
       config:
         stat_prefix: ext_authz
         grpc_service:
           envoy_grpc:
             cluster_name: patched.authservice.authservice.svc.cluster.local
           timeout: 300s # Increased this to 5 mins due to the issue described
---
# https://github.com/istio/istio/issues/21841
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authservice-cluster-patch
  namespace: kiali
spec:
  workloadSelector:
    labels:
      app: kiali
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        service: authservice.authservice.svc.cluster.local
    patch:
      operation: MERGE
      value:
        name: patched.authservice.authservice.svc.cluster.local

My authservice configuration looks like:

{
  "listen_address": "0.0.0.0",
  "listen_port": "10003",
  "log_level": "trace",
  "threads": 8,
  "chains": [
    {
      "name": "idp_filter_chain",
      "filters": [
      {
        "oidc":
          {
            "client_id": "[REDACTED]",
            "client_secret": "[REDACTED]",
            "authorization_uri": "https://login.microsoftonline.com/<APPLICATION_ID>/oauth2/v2.0/authorize",
            "token_uri": "https://login.microsoftonline.com/<APPLICATION_ID>/oauth2/v2.0/token",
            "jwks": "<REDACTED>",
            "callback_uri": "https://kiali.[REDACTED]/oidc/callback",
            "scopes": [],
            "id_token": {
              "preamble": "Bearer",
              "header": "Authorization"
            },
            "logout": {
              "path": "/authservice_logout",
              "redirect_uri": "https://kiali.[REDACTED]"
            }
          }
        }
      ]
    }
  ]
}

I am using the latest version of authservice (0.3.1) and of Istio (1.6.5).

kiali-proxy.log
authservice.log
authservice-proxy.log

[markszabo]

hi @dippynark did you figure out a solution to this?

I'm facing the same issue, and I really can't get my head around it. Setting the token URL to a site that logs the request, and then manually sending the request to login.microsoftonline.com works as expected 🤔

[dippynark]

@markszabo sorry I didn't in the end, but how did you do the request capture? Is it some open source tool? Sounds like a really strong way of debugging flows like this

I went for oauth2-proxy in the end (write-up here: https://www.jetstack.io/blog/istio-oidc/)

[markszabo]

Thanks for the suggestion, I'll take a look!

So for the capture I just set the token_uri to a https://mockbin.org/<my random id> URL, that logs the request. Then I reconstructed the request with burp suite (but probably Postman or curl would also do) and sent it manually to login.microsoftonline.com:

POST /<tenant id>/oauth2/v2.0/token HTTP/1.1
Host: login.microsoftonline.com
content-type: application/x-www-form-urlencoded
Content-Length: 1053
Authorization: Basic Y2xpZW50LWlkOmNsaWVudHMzY3IzdA==

code=Vm0wd2QyUXlVWGxWV0d4WFlURndUMVpzWkZOalJsWjBUVlpPV0Zac2JETlhhMk0xVmpBeFdHVkVRbUZTVjJoeVZteFZlRll5VGtWUmJGWlhZa2hDVVZadGNFZFRNbEpJVm10V1VtSklRazlVVkVKTFUxWmFjVkZ0UmxSTmF6RTFWVEowVjFaWFNraGhSemxWVm14YU0xWnNXbUZrUlRGVlZXeHdWMDFFUlRGV2EyUXdZekpHYzFOdVVsWmlhMHBZV1ZSR2QyRkdjRmRYYlVaclVsUkdXbGt3WkRSVk1rcFhVMnR3VjJKVVJYZFpWRVpyVTBaT2NtRkdXbWxTYTNCdlZtMXdUMVV4WkVkVmJGWlRZbFZhY1ZscmFFTlRiR3QzV2tSU1ZrMXJjRmhWTW5oelZqRmFObEZZYUZkU1JWcDZWbXBHVDJSV1ZuUmhSazVzWWxob1dGWnRNREZrTVVsNVZXNU9hbEp0VWxsWmJGWmhZMnhXY1ZKcmRGUlNiR3cxVkZaU1UxWnJNVVZTYTFwWFlsaENXRlpxUmtwbGJVWklZVVpvVjJKV1NrbFdiWEJIVkRKU1YxVnVVbXBTYkVwVVZteG9RMWRzV1hoWGJFNVRUVmQ0V1ZWdGRHdFdNV1JJWVVac1dtSkdXbWhaTW5oWFkxWkdWVkpzVGs1V01VbzFWbXBLTkZReFdsaFRiRnBZVmtWd1dGbHNhRU5oUmxweFVtMUdVMkpWVmpaWlZWcGhZVWRGZUdOSE9WZGhhMHBvVmtSS1QyUkdTbkpoUjJoVFlYcFdlbGRYZUc5aU1rbDRWMjVTVGxaRlNsaFVWbFY0VFRGU1ZtRkhPVmROVjFKSldWVmFjMWR0U2tkWGJXaFhUVVp3VkZacVNrZFNiRkp6Vld4a2FWZEdSalpXYlhCTFRVWlJlRmRZWkU1WFJYQllXVmR6TVZsV1VsWlhibVJYVW14d2VGVnRNVWRXTURGeVRsVm9WMUo2UmtoV1ZFWkxWakpPUmxac1pHbFNNVVYzVmxaU1IxbFdXbkpOVmxwWFlYcFdWRlZyVmtaT1VUMDk=&grant_type=authorization_code&redirect_uri=https%3A%2F%2Fmydomain%2Fproductpage%2Foauth%2Fcallback

And this request returned the expected result.

I also removed the client secret before setting the token_uri to https://mockbin.org/, and then put it back into the Authorization header before sending the request manually. Still this method exposes the OIDC response code to an external service, so only do this with test data (but actually the code can only be used once, so if you forward all the requests manually, then the codes become unusable).

[markszabo]

I managed to setup a proxy that terminates TLS and simply forwards the request to login.microsoftonline.com and that worked. My guess is that authservice doesn't like the certificate of login.microsoftonline.com, but this is just guessing (based on the fact that everything else is the same with the proxy setup).

Here is how I set up the proxy:

1. Download and install burp suite
2. Configure the proxy to listen on all interfaces, and enable Invisible mode. Export the CA certificate in DER format
   
3. Convert the certificate to pem: openssl x509 -in cacert.der -inform DER -out cacert2.pem -outform PEM
4. Replace the new lines with \n in the certificate to get something like this: -----BEGIN CERTIFICATE-----\nMIIH0DCCBrigAwIBAgIQCDHtasOWiQVFocXX2//+wjANBgkqhkiG9...w0BAQsFADBRb/5\n6tqE+0BOK1TBYR4scXNZO7CIgUyEISius6U2WYm98KIjt3uo\n-----END CERTIFICATE-----
5. Setup port forwarding on your home router to forward port 443 to port 8080 on the machine burp is running on
6. Get your public IPv4 address
7. Set the "trusted_certificate_authority" config option to the certificate from the previous step
8. Leave the "token_uri": "https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token"
9. For the authservice Deployment configure a hostAlias:

   template:
    metadata:
      labels:
        app: authservice
    spec:
      hostAliases:
      - ip: "1.2.3.4" # set it to your home IP
        hostnames:
        - "login.microsoftonline.com"
      volumes:

10. Make sure intercept is turned off in burp
    

Now it should work, and you should also see the requests and responses showing up in burp.