how to fix Azure app http status 400 substatus 604 error-

Question

how to fix Azure app http status 400 substatus 604 error-

华仲 25

Hello,

I deployed a web to Azure web app service a month ago. It frequently went down. I am asking how to fix this problem. Below is the detail.

the web app service is on windows server. I chose B1 as my service plan. My website is composed with a back-end api (.Net 6) and React SPA. In React, I used Axios to make HTTP request.

At the beginning, the website was fine. Myfriends and me were able to visit and interact with it. But later, we cannot visit the website. It frequently went down. In the Browser Debug mode Network tab, you could find the xhr request said" cannot load response data". But if you directly typed a api url (a "get" api in my website api) in the browser, it would give you the correct json data. Then the webiste suddenly became good. You could visit it normally. For about 30 mininutes later (the time is estimated), the webiste went down again.

The azure troubleshooting detects some errors. It said the front-end(the seven-load balancer functioning as proxy) detect some error regarding "Client Gone-The client that originated the request seems to have closed the connection before the response was received." . Furthermore, it said that Proxy exceptions detected where the client disappeared before the request could be completed. The diagnostic page also gave me a table displaying proxy exceptions that occurred while sending requests from the FrontEnd to the worker, resulting in exceptions.Here is the detail info in the table.

krpbfe: ProxyError, Site: blueprintforlife, Host: blueprintforlife.azurewebsites.net, Method: GET, Url: /api/getThumbnail, ServerRouted: 10.0.0.12, ProxyError: RequestCanceled, RequestAborted: True, RemoteAddress: 111.163.122.252, ConnectedTime: 14734ms, ProxyException: System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at Microsoft.Web.Hosting.Frontend.Utilities.ProxyMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageInvoker.g__SendAsyncWithTelemetry

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-21T10:41:49.43+00:00

@华仲 , Just checking in to see if you had got a chance to see the previous response. If the answer helped (pointed, you in the right direction) > please click Accept Answer Or please share the requested/more info to help you better.

Accepted answer

1 additional answer

Your answer

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-21T10:41:49.43+00:00

@华仲 , Just checking in to see if you had got a chance to see the previous response. If the answer helped (pointed, you in the right direction) > please click Accept Answer Or please share the requested/more info to help you better.

Answer 1

To benefit the community, posting the offline discussion:

As @华仲 mentioned :

"The website down symptom was the same no matter in plan D1 or in plan B1 (the app was in plan B1 in November the whole month). The only difference is if it is in plan B1 and turn on "always on", the issue is gone. "

Based on the webapp usage/requirement, the recommendation would be to leverage B1 plan with Always On feature enabled.

To provide more background on this, by default, web apps are unloaded if they're idle for a set period of time. This way, the system can conserve resources. The downside is that the response to the first request after the web app is unloaded is longer, to allow the web app to load and start serving responses. In Basic and Standard service plans, you can turn on the Always On setting to keep the app always loaded. This eliminates longer load times after the app is idle.

FAQ: How do I decrease the response time for the first request after idle time?
https://learn.microsoft.com/troubleshoot/azure/app-service/web-apps-performance-faqs#how-do-i-decrease-the-response-time-for-the-first-request-after-idle-time

华仲 25 Reputation points

2024-01-10T01:23:56.5033333+00:00

Plz copy paste your answer from the comment area to a new Anwer, and I will mark "accept answer" again on the new answer. I guess that way, the answer would be more obvious to readers.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-10T06:32:58.18+00:00

Thank you for your feedback. I have converted my latest comment to answer. Much appreciate your collaboration.

Answer 2

ajkuma 28,036 Microsoft Employee Moderator

@华仲 ,

Edit: Update from offline discussion, to benefit the community.
Based on the webapp usage/requirement, the recommendation would be to leverage B1 plan with Always On feature enabled.

Based on the information you provided, it appears that the issue is related to the client closing the connection before the response is received. This can happen due to various reasons such as network issues, server overload, or application errors.

I understand your WebApp was provisioned 1 month ago. If you haven’t done, Please take a look the steps to isolate the issue:

1.As a quick step to isolate the issue, you may scale up or scale-up the App Service Plan and see if it helps. (you may later scale-down)

Azure App Service represents a fleet of scale units. An app service always runs in an ASP.

I understand you have reviewed the diagnostic logs. Review Enable and review logs to fetch more details about the error.

Check if the issue is confined to any specific clients/users or happens to everyone.

If Always On is disabled, please enable it.

When Always On isn't turned on (default), the app is unloaded after 20 minutes without any incoming requests. The unloaded app can cause high latency for new requests because of its warm-up time. When Always On is turned on, the front-end load balancer sends a GET request to the application root every five minutes. The continuous ping prevents the app from being unloaded.

4.Turn off ARR affinity: If it’s a multi-instance deployment, ensure that the client is routed to the same instance for the life of the session. You may set this option to Off for stateless applications and then check how it goes.

Kindly let us know, I’ll follow-up with you further.

华仲 25 Reputation points

2023-12-21T13:45:28.9733333+00:00
Hello Kuma

I'm glad that someone from Azure reach out to me, otherwise I will feel frustrated without knowing what to do.

I did some attempts following your advice. Here is the findings:

Enabled logs and downloaded the logs. From the log, I found sql connection errors corresponding to the time I visited the website. I forgot to say in my original post, I used Azure SQL as my database. (The price-tier for the Azure SQL is General Purpose - Serverless: Standard-series (Gen5), 2 vCores. ) I put the error detail at the end of this post. To Fix this error, I revised my connection string, following the connection string Azure provided.

"Server=tcp:{host},1433;Initial Catalog={database};Persist Security Info=False;User ID={userid};Password={password};MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

Previously I didn't write "tcp: " before the host, also, I ommitted "MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"

After this revisement, I tried again. The connection error still appear in the log, corresponding the time I initiated the request from browser.

The website is still down. After typing a get api in the browser, the get api get successful result and the website is OK. About 25 minutes later, the website is down again. So the symptom is exactly the same as before.

I tried to turn off the ARR affinity in the Azure App--Configuration-GeneralSetting-Session affinity. The symptom was still the same. Nothing changed.

I tried to turn on the "Always on" in the Azure App--Configuration-GeneralSetting The sympton is gone! The website is up and I waited 30 min and tried again ,it is still up. I checked the log, there is no SQL connection error occurred.

So I guess a we can narrow down the issue a bit now. Would you please help me find the root problem and solve it? ( I do need to scale down my app Service to reduce the cost. I probably will downgrade from B1 plan to D plan and D plan does not have "always on "option. )

a) Do you think it has something to do with the Azure sql connection issue? For the Serverless Azure SQL I am using, the current "Auto-pause delay" in Settings--Compute+Storage is set to 1 hour.

Do you recommend I add some re-try logic to the sql connection code so that if the connection fails due to Azure SQL going to latency, it will initiate a 2nd try?

b) My web front-end is using React. The http request is using Axios. Here is the code . Shall I increase the timeout time from 5,000 ms to 50,000 ms?

const http = axios.create({ withCredentials: true, baseURL: 'XXXXXXXXXXXXXX', timeout: 5000 })

Please let me know your recommendations. Thank you!

Here is the detail error info in the diagnostic log.

2023-12-21 07:58:21.924 +00:00 [Error] Microsoft.EntityFrameworkCore.Database.Connection: An error occurred using the connection to database 'blueprint' on server 'blueprintdbs.database.windows.net,1433'.

2023-12-21 07:58:22.014 +00:00 [Error] Microsoft.EntityFrameworkCore.Query: An exception occurred while iterating over the results of a query for context type 'BluePrintBackEndApi.DatabaseContext.DBContext'.

Microsoft.Data.SqlClient.SqlException (0x80131904): Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=337; handshake=91; [Login] initialization=16; authentication=20; [Post-Login] complete=14056;

---> System.ComponentModel.Win32Exception (258): The wait operation timed out.

at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)

at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)

at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)

at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)

at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)

at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)

at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)

at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)

at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)

at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)

at Microsoft.Data.SqlClient.SqlConnection.Open()

at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerConnection.OpenDbConnection(Boolean errorsExpected)

at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)

at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)

at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)

at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.InitializeReader(Enumerator enumerator)

at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.<>c.<MoveNext>b__21_0(DbContext _, Enumerator enumerator)

at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult]

at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()

华仲 25

Hi Kuma,

Followed by my last comment, I supplimented with retry logic for both Database connection code and axios code. The Database connection retry is added in program.CS (.Net6) .The axios retry logic is also shown below.I'm sure the axios re-try logic is actually working since I moniter the browser network and saw indeed it tried 4 times in total.

But unfortunately, things didn't improve. The symtom is similar like before.

From the event log, I still see the same Database connection error like before.

So what would you recommend next?

import axios from "axios"
import axiosRetry from "axios-retry"
import { history } from "./history"
const http = axios.create({
  withCredentials: true,
  baseURL: window.location.origin + '/api', 
  timeout: 5000 
})
axiosRetry(http, {
  retries: 3, 
  retryDelay: () => 500, 
  shouldResetTimeout: true, 
  retryCondition: (error) => ['ECONNABORTED', 'ERR_NETWORK'].includes(error.code), 
})

builder.Services.AddDbContext<DBContext>(options =>
            
                options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection"), options =>
                {
                    options.EnableRetryOnFailure(); //Transient Error regarding connectivity-it will retry 6 times
                        
                })

            );

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-22T15:05:54.6933333+00:00

Thanks for the follow-up and sharing additional info. We wish to engage with you offline for a deeper investigation, please send an email with subject line “Attn:Ajay” to AzCommunity[at]Microsoft[dot]com, referencing this thread - I'll follow-up with you.
华仲 25 Reputation points

2023-12-23T09:23:05.9766667+00:00

I sent an email to AzCommunity@Microsoft.com, with subject line "Attn:Ajay", but the email cannot be deliverred due to "The group mtmvp only accepts messages from people in its organization or on its allowed senders list, and your email address isn't on the list."

Do you have other e-mails I can reach?
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-26T05:03:25.21+00:00

Apologies for the delayed response from over the weekend. I have received your email and responded. Thanks for your patience!

Share via

how to fix Azure app http status 400 substatus 604 error-

1 additional answer

Your answer