Our Azure App Service application started to experience intermittent CORS errors

Allan 290

Hi all,

I am new here, and I am hoping someone can help as we are completely at a loss with the problem we are experiencing suddenly with Azure App Service.

We have 2 Azure App Services, 1 for API and the other is an Web application (ReactJS). Both are hosted under different subdomain (api.domain.com and web.domain.com), and we manage our CORS and Referrer Policy via .NET Core Startup.cs, and not within Azure App Services. Both of these app services are hosted within the same App Service Plan (1 instance).

Our web application has been working for the past 3 years, but about a week ago our users started to report intermittent CORS errors. The error usually last about < 1min, and it goes away on its own, but it keeps coming back to different users every few minutes.

We have checked our recent releases but nothing seems to be the root cause this problem. Our team couldn't reproduce the CORS on our end (as it's quite random).

When we look at the Developer tool to analyze the HAR file, the only thing that looks suspicious is that the preflight call prior to the failed request always seem to be using port 80 for the remote address. We don't know whether this is the cause and why the browser would downgrade HTTPS to HTTP?

Our Referrer Policy is currently set as "strict-origin-when-cross-origin".

Wondering if anyone have experience something similar or if any experts have any suggestions on how to investigate to this problem we are having?

Any help is greatly appreciated.

Thank you in advance!

Allan

Allan 290 Reputation points

2024-05-24T20:47:26.5033333+00:00

Sorry, I should also mentioned that in the preflight call, "Access-Control-Allow-Origin" is present and with the proper URL. The failed request actually shows HTTP 500, but on the console error log is showing that there is a CORS error.

Could the CORS error be a false positive, and we are looking into the wrong place?
D4Dilip 30 Reputation points

2024-05-25T04:57:37.13+00:00

We were also facing the same issue yesterday.

our app was working fine for a year now and suddenly we started getting cors issue

after 2 hours of digging , we had to add our front end app domain name to cors ( explicitly ) along with *

i am sure this is some issue on azure side and its not communicated anywhere yet
Allan 290 Reputation points

2024-05-25T17:12:28.6533333+00:00

Hi @D4Dilip

Interesting. We were suspecting this could be Azure's problem as well, but unfortunately we are not sure how to report this.

Are you handling CORS via Startup.cs as well, or from the App Service CORS setting?

We will try to explicitly add the front end URL to CORS right now and see if the problem persist. Fingers crossed...

Thanks,

Allan
D4Dilip 30 Reputation points

2024-05-25T18:00:25.35+00:00

We have back end in nodejs and front end in react , running as different web app

the cors were set to * in app service until this issue came in yesterday

everything was working , no new release , no configuration changed at all

We had to add front end app base url https://app.domain.com to the backend api app service settings

its working so far :)
Andre Fuller 151 Reputation points

2024-05-29T15:47:47.1566667+00:00

We've been seeing the same issue. Incredibly random and will go away on it's own after a brief moment.

I'll try out adding our domain in addition to the wildcard and see if that resolves anything.

It does seem like it's something on Azure's side which is quite frustrating.
Allan 290 Reputation points

2024-05-29T17:01:42.0833333+00:00

Hi Andre,

Unfortuantely, the problem persists for us up until yesterday even after adding the full domain names to our CORS Policy. It was extremely frustrating, we have checked all logs / metrics and no errors or warning reported, and nothing suggests that we have any resource exhaustions. The only suspicious thing we found was that we discovered some network communication failures (3 way handshake error) between our App Service and our public IP.

Today, we redeployed the exact same API application (no CORS or any other change) under a new App Service Plan and App Service. No problem has been reported since, and so far it seems to be ok.

We suspect that there might be a network issue that hasn't been discovered.

We have a ticket opened with Microsoft since Monday, and I have shared this information with them. Hopefully, they can look into this further.

Hope this helps.
Daneel Olivaw 15 Reputation points

2024-05-29T17:15:42.6733333+00:00

We might have similar problems. Our web site has seen frequent request failures to our backend api in App Service in recent weeks - can't figure out why. From the webserver-log, it looked like the request was never received.

Andre, Allan, in which regions are you guys hosting your service?
ajkuma 24,396 Reputation points Microsoft Employee

2024-05-30T08:17:43.36+00:00

Apologies for the inconvenience and frustrating experience with this issue. I'm checking on this and will get back to you soon.

Allan, Kindly share the #SR details, I'll track this internally.
Allan 290 Reputation points

2024-05-30T13:13:49.38+00:00

@ajkuma our ticket number is 2405270040005953.

An extra set of eyes on this will be greatly appreciated. It has been almost a week since we submitted the ticket.
Allan 290 Reputation points

2024-05-30T13:25:40.63+00:00

@Daneel Olivaw we are in Canada East.
Fethi OUALI 10 Reputation points

2024-05-30T13:44:18.21+00:00

Hello,

We have a similar case here, region France.
Andre Fuller 151 Reputation points

2024-05-30T13:47:22.5133333+00:00

@Daniel Olivaw We have a traffic manager set up in front of three web app services, one in US East, one in US Central, and one in US West. We're not sure if it's one particular server or not. But like you, we're not even seeing the requests on our servers. What appears to be happening is the request goes to the Azure server, but the server throws a 500 response without it ever reaching our code. And that 500 response doesn't have the correct headers so in the console it shows as a CORS issue but really it's a random 500 error the server threw with no reason. I could be wrong on this but it's one theory I have. It's unfortunately hard to be sure when the issue happens so randomly.
Cecil Lew 25 Reputation points

2024-05-30T17:04:55.6833333+00:00

The symptoms described in this stackoverflow sound similar to ours. And it started in May 2024.
https://stackoverflow.com/questions/78551347/asp-net-core-web-api-on-azure-app-service-post-and-put-requests-failing-500-fo
Andre Fuller 151 Reputation points

2024-05-30T18:05:07.5266667+00:00

@Cecil Lew Thank you for the link, that's actually very helpful information.

We just got the error again and with the knowledge from that Stack Overflow question I can confirm the exact same thing is happening to us:

GET Requests work fine and don't have any issues

POST Requests will randomly get 500 errors that seemingly never make it to our code. The 500 error comes back with almost no headers so it omits the Access-Control-Allow-Origin header that we have configured on the server. That's why the console gives the CORS warning but in reality it's the server not even making it to our code and throwing a 500 error that we have no trace of whatsoever.
Allan 290 Reputation points

2024-05-30T19:01:38.1366667+00:00

Wondering if everyone is using HTTP/1.1 or HTTP/2 with your App Service (under configuration)?

We have been doing some A/B testings, and noticed that the only difference between our old App Service (HTTP/2) and the new one (HTTP/1.1) is this setting.

Once we changed our new backend App Service that has been working since yesterday to HTTP/2, our frontend application almost immediately started to report "Network Error" errors (still at random), and everything is back to normal when we switch back to HTTP/1.1.

Maybe Azure had some recent update or patch that affected their HTTP/2 configuration.

I am no expert in HTTP protocol, so here is some related information I found on ChatGPT.

https://chatgpt.com/share/8050da34-f08e-4916-a001-a57781cf2a3d
Scott M 220 Reputation points

2024-05-31T00:30:51.5733333+00:00

We are experiencing the exact same symptoms, only POST requests failing with intermittent CORS errors and no trace of the requests ever making it to the server in the api logs.

For the record we have a React SPA talking to .NET 8 web API with CORS set in code (not app service settings). We had it set to HTTP 2.0 but are now trying 1.1 to see if that resolves the issue.
Lars Cabrera 125 Reputation points

2024-05-31T07:55:43.9666667+00:00

Same issue here.

POST and PUT requests with JSON body and Content-Type: application/json fail randomly with no response code, response headers nor response body. No issues with GET. Interestingly enough, POSTs with Form Data seem to work just fine, even on hot-path endpoints. Maybe an issue with a JSON parser in a logging tool at Azure?

We’re in talks with Microsoft, but they haven’t found the issue yet.
Rick Baas 0 Reputation points

2024-05-31T12:21:18.77+00:00

We have the same issue. One of our engineers did notice something interesting. He decided to analyze the 500 responses by using Fiddler. Interestingly, when Fiddler was in use, the requests worked correctly. However, without Fiddler, the same 500 error persisted. We turned Fiddler on and off a couple of times and it kept working with Fiddler and stopped working without Fiddler. I don't have a lot of knowledge on networking but maybe this information is useful to someone.
Chan, Wellen 10 Reputation points

2024-05-31T13:17:07.3233333+00:00

We are also noticing the same issue with an API that we are hosting on an App Service.

We use a front-end app that communicates with the API with HttpClient (.NET 6) using HTTP/2 requests by default. Suddenly one of the instances in the front-end app started constantly receiving 500 errors from the API (POST calls).

Restarting the instance, seems to resolve the issue. However another instance began acting up and restarted that one too.

We hope to now "temporarily" migitated the issue by setting the HTTP Version of the API to HTTP/1.1. The front-end app is configured to fallback to HTTP/1.1 communication when API doesn't support it.

For now we're not having this issue...

Important to notice, is that our front-end app is also hosted on an App Service. However, we don't see the issue there. Perhaps because there is Cloudflare in front of it.
Khanh Tran 15 Reputation points

2024-06-04T04:13:18.8366667+00:00

We are experiencing the same issue in our production environment.
Yesterday, we tried a workaround by downgrading to HTTP/1.1, but it did not work.

Are there any other solutions to fix this?
Lars Cabrera 125 Reputation points

2024-06-04T08:55:15.5566667+00:00

Possible duplicate: https://learn.microsoft.com/en-us/answers/questions/1686913/randomly-post-calls-are-being-blocked-for-few-seco?page=1&orderby=Helpful&comment=answer-1533825#newest-answer-comment
Simon Opelt 6 Reputation points

2024-06-07T10:20:56.0366667+00:00

Just to add detail/cross reference: This issue seems to be (mostly) limited to Chromium 124/125-based browsers (Edge, Chrome ..) but does not seem to occur when using Firefox (against the same azure app service).

https://issues.chromium.org/issues/344633751
Patrick Schadler 55 Reputation points

2024-06-07T10:26:01.8+00:00

We got this on Firefox and Safari (Desktop/Mobile) as well.
Atle Magnussen 0 Reputation points

2024-06-07T10:48:29.73+00:00

@Simon Opelt I have reproduced the same scenario on Safari webkit.
But not Firefox, did not test for long though.
Ilias Michalarias 15 Reputation points

2024-06-08T10:29:24.8133333+00:00

We have also been experiencing the same issue for the past 2 weeks... Please let us know as soon as the fix is deployed across all Azure regions...
Michael Keymolen 0 Reputation points

2024-06-13T13:21:17.0066667+00:00

Now the fix has been implemented, is there a good reason why we should put our http version back on 2.0 instead of 1.1 ?
Jose Patino 65 Reputation points Microsoft Employee

2024-06-13T16:42:54.59+00:00

Http1.1 is slower than Http2.

8 answers

Jose Patino 65 Reputation points Microsoft Employee

2024-06-12T18:11:00.3066667+00:00

As of June 11, 2024 18:13 UTC the fix is pretty much completely done.

The fix goes by stamps so in order to know if your environment has the fix we would need to know the details of your environment to know in which stamp it is hosted.

One way to do it would be by creating a ticket and provide the information, that way we will know if your environment has been patched already or if is not we can patched. However as of 2024-06-11 about 88% of all the front ends are patched including nearly all multitenant.
Please sign in to rate this answer.

1 person found this answer helpful.
Ilias Michalarias 15 Reputation points

2024-06-13T21:37:03.9166667+00:00

Could you please inform us as soon as 100% has been reached? We seem to still be affected in one of our two systems... Thank you for your efforts!

Khanh Tran 15 Reputation points

2024-06-14T04:14:54.2133333+00:00

Where can I create a ticket? In Azure Portal Support or here? Our system is still experiencing issues even after downgrading to HTTP/1.1.

kjeske 11 Reputation points

2024-06-17T10:03:33.4+00:00

When can we expect the RCA? Where will it appear?

Jose Patino 65 Reputation points Microsoft Employee

2024-06-18T04:39:49.3166667+00:00

@khanh Tran all the Front Ends have been patched, so that particular issue is mitigated, however, if you are experiencing similar issues it can be a different issue altogether and I would recommend to create a ticket from the portal.
Sign in to comment
Jose Patino 65 Reputation points Microsoft Employee

2024-06-17T16:50:45.34+00:00
The Microsoft Azure Team has investigated the issue you reported defined by intermittent 500 errors starting around mid-May 2024 from PUT/POST HTTP requests for your Microsoft.Web resources.

Upon investigation, engineers determined that the behavior was caused by a race condition in our FrontEnd load balancers. This race condition was triggered by one of our standard maintenance upgrades to our FrontEnds.

As soon as the issue was identified the engineering team worked on a fix and immediately started rolling it out. The fix for this issue has now been fully rolled out and all impacted scale units are patched.

We are always aiming to improve our stability and reliability. In this case that includes, but is not limited to:

Code change to prevent this issue from happening again in the future

Improving our perf + testing capabilities to have a bigger focus on such scenarios

Enhancing our logging and monitoring systems to detect and alert us to any similar behavior as soon as possible

We sincerely apologize for any inconvenience this issue may have caused and appreciate your understanding as we continuously work to improve the reliability and performance of Azure services.

Regards,

The Microsoft Azure Team
Please sign in to rate this answer.
Nitish Devadiga 30 Reputation points

2024-06-17T16:57:31.2833333+00:00

Is this supposed to be the RCA?
Also, if an ongoing issue is identified, why is it not alerted to us via email or the Azure dashboard? We had spent many days and engineering hours identifying the cause of the issue only to discover in the end it was Azure's fault through this thread.

Andre Fuller 151 Reputation points

2024-06-17T17:03:45.2733333+00:00

Yeah this is entirely unacceptable. I'm sure there are still countless people that don't even know this occurred.

Patrick Schadler 55 Reputation points

2024-06-18T06:31:09.61+00:00

This is it? You can't be serious, folks. This is a MAJOR bug in one of your core services, and we didn't get an official statement apart from an answer on a random post?

This isn't a side-hustle where you expect some downtimes, this is a professional grade service and the lack of official communication is shocking and worrying at the same time.

Steve Tapley 5 Reputation points

2024-06-19T00:29:09.97+00:00

This disappeared for a few days, but I'm now seeing the error again - so I don't think this has been fixed properly.
Same pattern, a POST fails with a 500 error, never hits our server. All GETs work fine. Refreshing the page does not fix the issue. Trying in Incognito mode or another browser and it works first time.

Jose Patino 65 Reputation points Microsoft Employee

2024-06-19T00:52:03.3066667+00:00

@Steve Tapley I suggest to create a ticket so that your environment can be reviewed one on one, so far the previous issue has been resolved since all front ends have been patched
Sign in to comment
Vincent 5 Reputation points

2024-06-26T13:41:32.1533333+00:00

Hi, As of today I am still facing this issue with HTTP 504 errors.

Our App services are hosted in France Central, the frontend app service is in Angular 16 and the backend app service is in .NET 8.

We just tried the fix to change our HTTP version from 2 to 1.1.

I am concerned about this fix you mentioned since we still had A LOT of 504 errors even this very morning.
Please sign in to rate this answer.
Steve Tapley 5 Reputation points

2024-06-27T06:22:18.6333333+00:00

I had a similar issue - the issue reappeared. I had to open a new support incident with MS, and it was determined that the fix had not been applied properly to our instance.

Vincent 5 Reputation points

2024-06-27T07:48:50.1233333+00:00

I find it crazy that I have to buy a support plan to declare an incident that is none of our own doing. Do you know if there is any other way to open an incident ?
I am still having this issue with HTTP version 1.1 do you know if there is any other configuration that could solve this issue ?

Scott M 220 Reputation points

2024-06-27T07:58:11.4366667+00:00

If you are getting 504 errors and its occurring with HTTP 1.1 then my guess is that you probably do not have the same issue that was reported here. The common symptoms seemed to be intermittent periods with a response status of 0 on json put/post requests only (not get), http 2.0.

Vincent 5 Reputation points

2024-06-27T08:13:36.13+00:00

I have exactly the same behavior basically, it happens on GET/POST but it is intermittent, random, happens on some users but not all of them at the same time, if they switch browsers it can work instantly back again, we experienced CORS errors as well, etc. Even if it is not exactly the same, it looks very much alike I think.
But maybe you're right and the problem or the fix is not exactly the same since we have 504 errors.
Sign in to comment

Share via

Our Azure App Service application started to experience intermittent CORS errors

8 answers