Certificate Order issue when using Azure App Gateway

[dtx3k]

Hi,

We are having difficulties with some application (not build by our selfes) that cannot connect to the Application Gateway
The error shown in the application is that the certificate order is incorrect

investigating this issue using SSLlabs for example do indeed show that the order is incorrect indeed

The certificates are LetsEncrypt issued using this awesome keyvault-acmebot and stored in keyvault
Azure Application gateway retrieves the certificate from the keyvault and uses is for its HTTP Listener

Using a browser for testing does not show that the order is incorrect and accepts it as it is, as long that it is complete

We opened a support case at Microsoft to see if we can fix the issue, unfortunatly they only have a tool that manipulates the cert to correct the order in keyvault. this is a manual process, as you would understand this is not a viable sollution in a fully automated system.

TL;DR:
My question is; Is there a way to manipulate the Cert Order before storing it to the keyvault ?

Error in application:

Error in SSLlabs:

[shibayan]

Is the Application Gateway v1 or v2? I tried to create a v2 environment, but I can't seem to reproduce it.

[dtx3k]

it is a V2

Certbot > KV > AppGW-V2
Not something you setup in 5 min :) (AppGW can be overwhelming)

[shibayan]

I was able to reproduce it and will fix it in #407. Thanks.

[dtx3k]

Awesome! thanks you so much

[shibayan]

Fixed in v3.7.2

[dtx3k]

wow thats fast! thank you, cant wait to test it
any idea when the TF module is also up to date? no rush!

Edit: nevermind, got it going bij restarting the app > renew cert, wait for AppGW to pick it up:

Fix confirmed! thank you so much for this fast resolve!!!

[snbingham]

Hi,

I'm seeing a similar issue after attempting to renew certificates today. Java wasn't happy with the order in the certificate file.

I downloaded the currently working PFX and used OpenSSL to convert the working crt file. It has:

Private Key
CA Cert:
subject=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3
CA Cert:
subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
Actual cert

When I download the PFX from the new cert and use OpenSSL to convert it I got;

Private Key
CA Cert:
subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
CA Cert:
subject=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3
Actual cert

Java would accept the new cert as valid if I swapped the two CA Certs round. We are using the latest.zip so we should have the fix outlined above. Could there be another issue?

Neil.

[dtx3k]

Hi,
not shure if you manage the java code or if its a third party application which is Java based
If you do manage the code maybe work with the keyvault in a different way then using download PFX
You could for example interact with the API like this: https://docs.microsoft.com/en-us/java/api/overview/azure/security-keyvault-certificates-readme?view=azure-java-stable

this way your in control and the way you work with Azure components and certbot wont break, and stay with current and future updates of Certbot

just my 2 cents :)

[shibayan]

Older versions of the application zip are not provided. You can use any version by cloning and deploying this repository

[shibayan]

@dtx3k @snbingham

The default behavior has been changed in v3.7.3.

• By default, the order of certificates has been updated to be the same as before the change in v3.7.2.
• Added an option for cases where the order of the certificates matters, such as Application Gateway.
  • If you set Acmebot:MitigateChainOrder to true, you can avoid incorrect order in the Application Gateway.

[dtx3k]

I can work with that :) thanks

[snbingham]

@shibayan many thanks for doing that, and so quickly. I can confirm newly issued certs with 3.7.3 are working for Java.

@dtx3k it is ElasticSearch that was complaining. Having checked the error I found the crt file ordering issue in the Elastic forums and confirmed what was happening on our setup. It's happy with the 3.7.3 issued certs.

[ayronjoj]

Hi, I'm facing the exact same problem. I
I've already set Acmebot:MitigateChainOrder to true, but this error order is incorrect with the error certificate invalid common name (CN).
Is there anything else I need to do or other parameters to activate this Acmebot:MitigateChainOrder ?
Obs: I've redeployed the whole app functions and re-created it using the azure template available on read-me

My case:
I'm using app gw and have a keylock (that needs to have end-to-end tls) running inside de cluster aks, I configured the aks to sync the cert generated from key vault to a secret on aks automatically, but when I export the azure key vault cert, the first chain is from the Let's encrypt, then the intermediate cert and last the URL chain itself. Compared to a scenario that works using a cert manager inside the cluster, the first chain is from the URL.
I've manual change de order and it worked, but if I can have this in order automatically would be awesome and fully resolve all my issues with renewing certificates.

[Kapsztajn]

@shibayan
I have the same issue as Ayronjoj. Basically Im generating certificates, use them in FrontDoor but I also wanna to use them in AKS. I use tool (akv2k8s) to copy certificate from AKV to Kubernetes secret and Im getting error from Nginx:

Error obtaining X.509 certificate: unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key

When I manually checked the certificate from Secret I could see that it is in order:
R3 -> ISRG Root X1 -> Domain
But I need this to start from Domain.
I added Acmebot:MitigateChainOrder to true, restarted the Function, renewed every certificate and nothing worked. Is there something more that Im missing?

[ayronjoj]

@Kapsztajn, I solved my problem using the Acmebot:MitigateChainOrder=true, set to reuse key on renew certificate when creating the cert , but the most important thing for me was adding "chainOrder: ensureserverfirst" on the akv2k8s manifest.
Find the akv2k8s manifest as an example below:

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: certificate-sync
  namespace: ${NS}
spec:
  vault:
    name: ${AZURE_KEYVAULT_NAME}
    object:
      name: wildcard-${TLD}
      type: certificate
  output:
    secret:
      name: wildcard-${TLD}
      type: kubernetes.io/tls
      chainOrder: ensureserverfirst

[Kapsztajn]

@ayronjoj Thank you so much for this info!

[culpinnis]

Hi,
I am having the same issue. I tried to activate Acmebot:MitigateChainOrder, but it does not make any difference 😞
I try to use the certificate outside of K8s on a VM deployed by Terraform with Cloudinit running nginx. I also get exactly the same error message when nginx is starting and I noticed the wrong order of the chain. Any advices?