What is wrong with our Azure Front Door CDN? Bad Gateway response caused by certificate issues?

Question

What is wrong with our Azure Front Door CDN? Bad Gateway response caused by certificate issues?

Hans 0

We have an Azure Front Door CDN that handles all our static resources for our website. Everything has been working fine for a year or so until yesterday. A request to our front door, e.g. https://cdn.example.com/image.jpg now results in a 502 Bad Gateway response and the message "Our services aren't available right now. We're working to restore all services as soon as possible. Please check back soon." A request direct to the same resource on our actual website, e.g. https://www.example.com/image.jpg, is working fine. Debugging by including the http header "X-Azure-DebugInfo: 1" gives some more information: "X-Azure-Externalerror: CertificateExpired". But which certificate has expired? The certificate for the origin website is up-to-date and everything seems fine in the Azure Portal, green checkmarks everywhere and no warnings. Any ideas what is causing this? What can we do to resolve it? Thanks

GitaraniSharma-MSFT 50,197 Reputation points Microsoft Employee Moderator

2024-02-20T11:29:11.1566667+00:00
Hello @Hans ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I understand that a request to your Azure Front Door is resulting in 502 Bad Gateway response with an error message "Our services aren't available right now. We're working to restore all services as soon as possible. Please check back soon", but a direct request to the backend origin is working fine. On further debugging, you found the following error: "X-Azure-Externalerror: CertificateExpired".

The error message is very clear from the debug log. X-Azure-Externalerror: CertificateExpired means the origin certificate is expired.

However, you mentioned that the certificate for the origin website is up-to-date, and everything seems fine in the Azure Portal with green checkmarks everywhere and no warnings.

I have seen many such cases, where one of the websites on the backend VM didn't have SSL updated or there was an Application gateway between the origin & AFD and one of the listener certificates expired.

So, I would request you to validate your setup and configuration again to confirm that there is no origin or service in between whose certificate has expired and needs rotation.

Also, do share the below details:

What is configured as your AFD origin? Is it a VM or a web app?

Do you have Application gateway or any other service behind the Azure Front Door?

Regards,

Gita
Priya Kumar 1,096 Reputation points Microsoft Employee

2024-02-20T11:41:45.87+00:00
Hello @hanshans ,

Thanks for reaching Azure Q and A platform.

Could you please follow the below steps to troubleshoot this issue further?

When you get the error, could you please paste the ref ID you see below the error message "Our services aren't available right now."??? That would help to get more information on this issue.

We need to understand if the Backend has provided 502 or the Front door.

Please do run the below curl command so that we get more information on the error:

curl -I -H "X-Azure-Debuginfo: 1" https://cdn.example.com/

This will give more information if the issue is with the Certificate on the Backend.

Also would be great to take a F12 log in the working and non-working scenario.

Please check if your "hostname" configuration is an IP address. If so then AFD will not set an SNI, indeed this will return you the default Certificate, mostly that certificate would have been expired.

If, your hostname is an IP address, try this Openssl command to get the default certificate:

~$ openssl s_client -connect IPaddress:443 -noservername -showcerts </dev/null 2> /dev/null | openssl x509 -text -noout -dates

If so, then please update this certificate to fix the issue.

Regards, Priya Kumar
Hans 0 Reputation points

2024-02-20T17:00:03.7533333+00:00

Hello and thank you for your replies! We found that the cause of the issue was an incorrect (previously working) origin host name. We have now updated to the correct host name. Do you recommend us to do a full purge? Regards

2 answers

Your answer

GitaraniSharma-MSFT 50,197 Reputation points Microsoft Employee Moderator

2024-02-20T11:29:11.1566667+00:00

Hello @Hans ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I understand that a request to your Azure Front Door is resulting in 502 Bad Gateway response with an error message "Our services aren't available right now. We're working to restore all services as soon as possible. Please check back soon", but a direct request to the backend origin is working fine. On further debugging, you found the following error: "X-Azure-Externalerror: CertificateExpired".

The error message is very clear from the debug log. X-Azure-Externalerror: CertificateExpired means the origin certificate is expired.

However, you mentioned that the certificate for the origin website is up-to-date, and everything seems fine in the Azure Portal with green checkmarks everywhere and no warnings.

I have seen many such cases, where one of the websites on the backend VM didn't have SSL updated or there was an Application gateway between the origin & AFD and one of the listener certificates expired.

So, I would request you to validate your setup and configuration again to confirm that there is no origin or service in between whose certificate has expired and needs rotation.

Also, do share the below details:

What is configured as your AFD origin? Is it a VM or a web app?

Do you have Application gateway or any other service behind the Azure Front Door?

Regards,

Gita
Priya Kumar 1,096 Reputation points Microsoft Employee

2024-02-20T11:41:45.87+00:00

Hello @hanshans ,

Thanks for reaching Azure Q and A platform.

Could you please follow the below steps to troubleshoot this issue further?

When you get the error, could you please paste the ref ID you see below the error message "Our services aren't available right now."??? That would help to get more information on this issue.

We need to understand if the Backend has provided 502 or the Front door.

Please do run the below curl command so that we get more information on the error:

curl -I -H "X-Azure-Debuginfo: 1" https://cdn.example.com/

This will give more information if the issue is with the Certificate on the Backend.

Also would be great to take a F12 log in the working and non-working scenario.

Please check if your "hostname" configuration is an IP address. If so then AFD will not set an SNI, indeed this will return you the default Certificate, mostly that certificate would have been expired.

If, your hostname is an IP address, try this Openssl command to get the default certificate:

~$ openssl s_client -connect IPaddress:443 -noservername -showcerts </dev/null 2> /dev/null | openssl x509 -text -noout -dates

If so, then please update this certificate to fix the issue.

Regards, Priya Kumar
Hans 0 Reputation points

2024-02-20T17:00:03.7533333+00:00

Hello and thank you for your replies! We found that the cause of the issue was an incorrect (previously working) origin host name. We have now updated to the correct host name. Do you recommend us to do a full purge? Regards

Answer 1

Hello @Hans ,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll repost your solution in case you'd like to "Accept" the answer.

Issue: A request to your Azure Front Door is resulting in 502 Bad Gateway response with an error message "Our services aren't available right now. We're working to restore all services as soon as possible. Please check back soon", but a direct request to the backend origin is working fine. On further debugging, you found the following error: "X-Azure-Externalerror: CertificateExpired".

Solution: You found the cause for the issue. It turned out the Front Door origin was incorrect, an old domain name whose certificate expired recently, but had been working up until now. After changing to the correct domain, you can confirm that the CDN URLs are working.

Answering your follow-up question below:

Do you recommend us to purge everything before start using the Front Door again?

Yes, it would be best to purge everything before start using the Front Door again.

If you have any other questions or are still running into more issues, please let me know.

Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Answer 2

Hans 0

Hello, and thank you for your replies. We have found the cause for the issues. It turned out the Front Door origin was incorrect, an old domain name whose certificate expired recently, but had been working up until now. After changing to the correct domain we can confirm that the cdn urls are working. Do you recommend us to purge everything before start using the Front Door again?

0 comments

Share via

What is wrong with our Azure Front Door CDN? Bad Gateway response caused by certificate issues?

2 answers

Your answer