Intermittent 503s on Azure Front Door from "OriginInvalidResponse"

Question

Intermittent 503s on Azure Front Door from "OriginInvalidResponse"

Don Wise 25

I'm using Azure Front Door as the load balancer for several services behind it. One in particular is a Wordpress sitting on a custom Linux VM. I also use Azure Domain services, and Azure Front Door manages the SSL certs. I see 1-2% of my traffic failing to 503s, and I hit it a lot myself when using my website heavily. I cannot for the life of me fix it. Uptime on all the back ends is 100%, leading me to see this as an Azure Front Door issue. The 503s are sparse, but it can be very jarring to the end user when it happens. Worse, this error page isn't even customizable.

I've scoured the internet for solutions and tried everything I could find, including:
https://learn.microsoft.com/en-us/azure/frontdoor/troubleshoot-issues#503-or-504-response-from-azure-front-door-after-a-few-seconds
https://learn.microsoft.com/en-us/answers/questions/955337/azure-front-door-returns-intermittent-503-response

The last article mentions that the Azure Front Door team helped fix something on their end, so I am unsure if I may need a similar fix.

I've used the following query to look at my logs and found that most of these issues are "OriginInvalidResponse." In the above suggestions, I've walked through the above and tried everything suggested already. Asking for help is my last resort.

AzureDiagnostics
| extend Is5XX = (toint(httpStatusCode_s ) >= 500 and toint(httpStatusCode_s ) < 600)
| where Is5XX == true
| project TimeGenerated, requestUri_s, httpMethod_s, httpStatusCode_s, rulesEngineMatchNames_s, cacheStatus_s, errorInfo_s, originName_s, originUrl_s, domain_s
| order by TimeGenerated desc 
| limit 100

Don Wise 25 Reputation points

2023-07-14T03:33:48.95+00:00

Also worth noting, from the URLS failing, I don't see a discernable pattern. It is random from what I appear. I've match the time stamps to the load on the system and also VM resources and found no correlation, it is not due to the system being stressed.
Don Wise 25 Reputation points

2023-07-14T03:36:19.73+00:00

<removed> was a delay in my prior comment being posted and thought it was lost.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-14T08:59:04.3866667+00:00

Hello @Don Wise ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I understand that you are receiving intermittent 503 error on Azure Front Door with code "OriginInvalidResponse".

Intermittent 503 errors with "ErrorInfo: OriginInvalidResponse" are mostly caused because the backend has a KeepAlive timeout less than 90 seconds. When the origin has a lower idle timeout that AFD's, the 503 errors are random and low volume.

It can happen if your backend closes a kept alive HTTP connection, right at the moment when AFD reuses the same connection for a new request.

Let's say your backend has an HTTP keepalive timeout that is less than 90 seconds. Azure Frontdoor reuses connections to improve performance, so when a connection is created for handling one request, that TCP connection is kept open for reuse (HTTP keepalive). AFD has an idle keepalive timeout of 90 seconds. But if your origin times out and disconnect sooner than this 90 second, then there can be a race condition that may result in this error. Specifically, AFD may reuse a connection, sending a new HTTP request, right at the moment when the origin times out and sends a TCP FIN. AFD interprets that receiving TCP FIN after sending a new request as an invalid response, and hence the error.

Unfortunately, this 90 second idle timeout is not configurable at AFD side.

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#front-door-to-application-back-end

To fix this issue, your backend should hold already used connections open for at least 91 sec, so AFD can reuse them for subsequent requests (if any). You need to make sure that the keepalive timeout on backend is more than 90 sec.

How is your backend configured? Will you be able to enable/change keepalives on your backend?

Regards,

Gita
Don Wise 25 Reputation points

2023-07-14T09:09:47.1233333+00:00

I see, so I should set the origin response timeout to 90 seconds and the backend to something larger. Let me give that a try.
Don Wise 25 Reputation points

2023-07-14T09:21:37.42+00:00

I have a custom Linux VM that I shell into for the back end, so it is configurable. I set it to 91 seconds keep alive. I also set AFD's Origin Response Timeout to 90 seconds. Let's give it some time and see if that was the magic.
Thank you for the help! Will respond tomorrow and crossing my fingers the 500 log is empty!

Also just noting the additional behavior I was seeing prior incase it helps others. All the 503s always returned ~.1 second or less which made it unusual. Perhaps this was due to connection reuse.
Don Wise 25 Reputation points

2023-07-15T04:16:20.3066667+00:00

I am happy to report it does seem significantly improved. I suspect there might still be issues, so I'll monitor over the next few days. But it does look much better so far—still some spikes.
Don Wise 25 Reputation points

2023-07-15T07:29:32.3833333+00:00

Unfortunately, I am still hitting this, comes and goes intermittently. Is there anything incorrect with what I mentioned above? The issue is not fixed.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-17T15:27:30.3866667+00:00

@Don Wise , since the issue is intermittent, it would require a deeper investigation by digging into the backend logs. So if you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.
Don Wise 25 Reputation points

2023-07-17T20:13:39.71+00:00

I don't have a support plan, I generally can handle most technical issues individually but as mentioned I have tried to fix this in every way I could think about from my end without success.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-18T15:17:34.6666667+00:00
@Don Wise , please send an email with subject line "ATTN gishar | Intermittent 503s on Azure Front Door with OriginInvalidResponse" to AzCommunity[at]Microsoft[dot]com with the following details, I will follow-up with you.

Reference this Q&A thread

Your Azure Subscription ID

Note: Do not share any PII data as a public comment.

We will post a summarized answer once the issue is resolved.

Regards,

Gita
Don Wise 25 Reputation points

2023-07-21T20:00:28.5833333+00:00

Good news, I was responding but observed I made a mistake with the http.d config. I changed the "Timeout" config but not the "KeepAliveTimeout," which is different. I did correct this and gave it two days, and it does appear fixed! I wanted to output my config below in case it's helpful for others.

The configuration for my AFD is "Origin Response timeout: 90 seconds".

And the http.d conf of the Apache server:

Thank you!
Don
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-21T21:08:46.26+00:00

@Don Wise , thank you for the update. Glad to hear that the issue is now resolved.

I have summarized our conversation as an answer below with the screenshot of the httpd.conf file of your Apache server for better visibility.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Accepted answer

0 additional answers

Your answer

Don Wise 25 Reputation points

2023-07-14T03:33:48.95+00:00

Also worth noting, from the URLS failing, I don't see a discernable pattern. It is random from what I appear. I've match the time stamps to the load on the system and also VM resources and found no correlation, it is not due to the system being stressed.
Don Wise 25 Reputation points

2023-07-14T03:36:19.73+00:00

<removed> was a delay in my prior comment being posted and thought it was lost.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-14T08:59:04.3866667+00:00

Hello @Don Wise ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I understand that you are receiving intermittent 503 error on Azure Front Door with code "OriginInvalidResponse".

Intermittent 503 errors with "ErrorInfo: OriginInvalidResponse" are mostly caused because the backend has a KeepAlive timeout less than 90 seconds. When the origin has a lower idle timeout that AFD's, the 503 errors are random and low volume.

It can happen if your backend closes a kept alive HTTP connection, right at the moment when AFD reuses the same connection for a new request.

Let's say your backend has an HTTP keepalive timeout that is less than 90 seconds. Azure Frontdoor reuses connections to improve performance, so when a connection is created for handling one request, that TCP connection is kept open for reuse (HTTP keepalive). AFD has an idle keepalive timeout of 90 seconds. But if your origin times out and disconnect sooner than this 90 second, then there can be a race condition that may result in this error. Specifically, AFD may reuse a connection, sending a new HTTP request, right at the moment when the origin times out and sends a TCP FIN. AFD interprets that receiving TCP FIN after sending a new request as an invalid response, and hence the error.

Unfortunately, this 90 second idle timeout is not configurable at AFD side.

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#front-door-to-application-back-end

To fix this issue, your backend should hold already used connections open for at least 91 sec, so AFD can reuse them for subsequent requests (if any). You need to make sure that the keepalive timeout on backend is more than 90 sec.

How is your backend configured? Will you be able to enable/change keepalives on your backend?

Regards,

Gita
Don Wise 25 Reputation points

2023-07-14T09:09:47.1233333+00:00

I see, so I should set the origin response timeout to 90 seconds and the backend to something larger. Let me give that a try.
Don Wise 25 Reputation points

2023-07-14T09:21:37.42+00:00

I have a custom Linux VM that I shell into for the back end, so it is configurable. I set it to 91 seconds keep alive. I also set AFD's Origin Response Timeout to 90 seconds. Let's give it some time and see if that was the magic.
Thank you for the help! Will respond tomorrow and crossing my fingers the 500 log is empty!

Also just noting the additional behavior I was seeing prior incase it helps others. All the 503s always returned ~.1 second or less which made it unusual. Perhaps this was due to connection reuse.
Don Wise 25 Reputation points

2023-07-15T04:16:20.3066667+00:00

I am happy to report it does seem significantly improved. I suspect there might still be issues, so I'll monitor over the next few days. But it does look much better so far—still some spikes.
Don Wise 25 Reputation points

2023-07-15T07:29:32.3833333+00:00

Unfortunately, I am still hitting this, comes and goes intermittently. Is there anything incorrect with what I mentioned above? The issue is not fixed.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-17T15:27:30.3866667+00:00

@Don Wise , since the issue is intermittent, it would require a deeper investigation by digging into the backend logs. So if you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.
Don Wise 25 Reputation points

2023-07-17T20:13:39.71+00:00

I don't have a support plan, I generally can handle most technical issues individually but as mentioned I have tried to fix this in every way I could think about from my end without success.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-18T15:17:34.6666667+00:00

@Don Wise , please send an email with subject line "ATTN gishar | Intermittent 503s on Azure Front Door with OriginInvalidResponse" to AzCommunity[at]Microsoft[dot]com with the following details, I will follow-up with you.

Reference this Q&A thread

Your Azure Subscription ID

Note: Do not share any PII data as a public comment.

We will post a summarized answer once the issue is resolved.

Regards,

Gita
Don Wise 25 Reputation points

2023-07-21T20:00:28.5833333+00:00

Good news, I was responding but observed I made a mistake with the http.d config. I changed the "Timeout" config but not the "KeepAliveTimeout," which is different. I did correct this and gave it two days, and it does appear fixed! I wanted to output my config below in case it's helpful for others.

The configuration for my AFD is "Origin Response timeout: 90 seconds".

And the http.d conf of the Apache server:

Thank you!
Don
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-21T21:08:46.26+00:00

@Don Wise , thank you for the update. Glad to hear that the issue is now resolved.

I have summarized our conversation as an answer below with the screenshot of the httpd.conf file of your Apache server for better visibility.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Answer 1

Hello @Don Wise ,

I understand that you are receiving intermittent 503 error on Azure Front Door with code "OriginInvalidResponse".

Intermittent 503 errors with "ErrorInfo: OriginInvalidResponse" are mostly caused because the backend has a KeepAlive timeout less than 90 seconds. When the origin has a lower idle timeout that AFD's, the 503 errors are random and low volume.

It can happen if your backend closes a kept alive HTTP connection, right at the moment when AFD reuses the same connection for a new request.

Let's say your backend has an HTTP keepalive timeout that is less than 90 seconds. Azure Frontdoor reuses connections to improve performance, so when a connection is created for handling one request, that TCP connection is kept open for reuse (HTTP keepalive). AFD has an idle keepalive timeout of 90 seconds. But if your origin times out and disconnect sooner than this 90 second, then there can be a race condition that may result in this error. Specifically, AFD may reuse a connection, sending a new HTTP request, right at the moment when the origin times out and sends a TCP FIN. AFD interprets that receiving TCP FIN after sending a new request as an invalid response, and hence the error.

Unfortunately, this 90 second idle timeout is not configurable at AFD side.

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#front-door-to-application-back-end

To fix this issue, your backend should hold already used connections open for at least 91 sec, so AFD can reuse them for subsequent requests (if any). You need to make sure that the keepalive timeout on backend is more than 90 sec.

I asked you to validate your backend configuration and enable/change keepalives on your backend to more than 90 seconds.

You have a custom Linux VM in your backend, and it is configurable. So, you set the "KeepAliveTimeout" to 92 seconds in the httpd.conf configuration file.

Refer: https://httpd.apache.org/docs/2.4/mod/core.html#keepalive

The httpd.conf file of your Apache server is now configured as below:

enter image description here

You waited for 2 days to observe the behavior and have confirmed that the issue is now fixed for you.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Share via

Intermittent 503s on Azure Front Door from "OriginInvalidResponse"

0 additional answers

Your answer