Our Azure App Service application started to experience intermittent CORS errors

Question

Hi all,

I am new here, and I am hoping someone can help as we are completely at a loss with the problem we are experiencing suddenly with Azure App Service.

We have 2 Azure App Services, 1 for API and the other is an Web application (ReactJS). Both are hosted under different subdomain (api.domain.com and web.domain.com), and we manage our CORS and Referrer Policy via .NET Core Startup.cs, and not within Azure App Services. Both of these app services are hosted within the same App Service Plan (1 instance).

Our web application has been working for the past 3 years, but about a week ago our users started to report intermittent CORS errors. The error usually last about < 1min, and it goes away on its own, but it keeps coming back to different users every few minutes.

We have checked our recent releases but nothing seems to be the root cause this problem. Our team couldn't reproduce the CORS on our end (as it's quite random).

When we look at the Developer tool to analyze the HAR file, the only thing that looks suspicious is that the preflight call prior to the failed request always seem to be using port 80 for the remote address. We don't know whether this is the cause and why the browser would downgrade HTTPS to HTTP?

Our Referrer Policy is currently set as "strict-origin-when-cross-origin".

Wondering if anyone have experience something similar or if any experts have any suggestions on how to investigate to this problem we are having?

Any help is greatly appreciated.

Thank you in advance!

Allan

Answer 1

We created the Stackoverflow post and thankfully was linked to this site.

(https://stackoverflow.com/questions/78551347/asp-net-core-web-api-on-azure-app-service-post-and-put-requests-failing-500-fo)

An update from us:

• The Http 1.1 temporary workaround seem to work so far.
• @Lars Cabrera I can confirm your findings from our end:
  • POST / PUT with application/json will fail all the time.
  • GET, DELETE and POST Formdata will work.
• Our fallback plan is also to port the POST to GET where we can and use Querystring instead. That leaves the POST/PUT that does updates which can be ported to use forms instead, but that will be last resort if this issue isn't fixed.
• The problem seems to be global and we've experienced in several regions.

We're thankful for everybody who is sharing in this thread. This seems very much a problem introduced by Microsoft and hopefully will be fixed soon.

Answer 2

This case has been escalated to the product group where it is been analyzed so that a fix can be released, will post back when we have RCA and fix is been rolled out

Answer 3

Posting our current findings here, as there seems to be a lot of confusion and outdated information flying around both this thread, duplicates and other sites.

• The intermittent errors seem to be triggered solely by requests to azure app services, containing a JSON body and the header Content-Type: application/json. The HTTP method does not seem to matter. POSTs/PUTs fail just the same with a JSON body, but not with form data. We have no confirmations for PATCH yet, as we have very little PATCH traffic, but we suspect it should behave just the same.
• Preflight (OPTIONS) calls are working as expected, but the POST/PUT calls are somehow rejected, without returning a single response header, HTTP status code 0, and empty body. This looks different in each browser, with some browsers supplying default values for status code (500) and Date (new Date()).
• The exact same calls will fail in the browser for a period of time, no matter if you hard refresh, empty cache, cookies and local storage, even if you copy the request as fetch and run it in the browser's console. However, in another browser, or if you kill the browser, the problem disappears.
• Nothing appears in any log in Azure.

We've tried the following:

1. Restaring app services
2. Replacing instances (scaling up/down)
3. Creating entirely new app services, in new resource groups, new locations, new app service plans
4. Downgrading from HTTP/2 to HTTP/1.1
5. Cache-breaking the CORS headers with Access-Control-Max-Age: 1

All attempts (except 5, which we implemented with application code and downtime-less deploy) cause a form of restart of the app services, which has shown to temporarily mitigate the issue. However, the problem comes back after a couple of hours, so it doesn't really solve anything. We cannot keep taking our applications down for restarts every couple of hours.

The only thing that has proven to work, is rewriting endpoints to either GET requests with query params, or POST/PUT with form data. JSON is broken. Maybe a memory leak in a built in logging tool in Azure? No idea, but the only commonality is JSON.

We've set up frontend (browser) error logging to catch the occurrence, and highly suggest you all do the same, if you have not already done so.

Rewriting our 200+ endpoints taking JSON as input to form data or query params is not a realistic strategy, but we're running out of ideas.

Answer 4

Hi Wellen, we too have temporary changed our AppService from HTTP/2 back to HTTP/1.1 and the problem has gone away since.

@ajkuma I have shared this information w/ MS support team, under the ticket # 2405270040005953. We haven't heard back with any other updates for a while. Perhaps, you can help follow up and escalate this, and share w/ them that there are others who are affected as well.

Answer 5

I am happy to announce that we have identified the issue and the testing has been validated.

The deploy with the fix has started.

Currently the workaround is to disable http2 on the app. This fixes it in almost all cases.