How to fix Azure Storage Account OperationTimedOut error

Malik, Deepankar 25

Our function app uses an Http Trigger to upload files to an ADLS storage account. During a recent load test, we encountered a StatusCode - 500 error. Upon investigation, we found that the storage account was timing out requests with an OperationTimedOut error and StatusCode - 500.

The storage account is a general-purpose v2 type located in eastus2. I reviewed some of the storage account metrics:

Transactions by storage type: The count ranged between 45K to 55K transactions during the period when we experienced the 500 status errors.
Ingress (sum): The data ingress was 161.2GiB during the same duration.

Sumarigo-MSFT 43,801 Reputation points Microsoft Employee

2024-04-16T07:52:26.5466667+00:00

@Malik, Deepankar For better understanding the issue: Can you please share the screen shot of the error message and request ID?

Can you please elaborate bit more on your query?

Based on the error message: The operation could not be completed within the permitted time. The operation may or may not have succeeded on the server side. Please query the server state before retrying the operation.

Thers is similar thread in Q&A Forum, please refer to the suggestion mentioned here

Additional information:

Based on the information you provided, it seems that the storage account is experiencing timeouts and 500 status errors during periods of high transaction volume and data ingress. Here are a few suggestions to help resolve this issue:

Increase the timeout value for the storage account: You can increase the timeout value for the storage account to allow more time for requests to complete. You can do this by setting the ConnectionTimeout property on the storage client object in your function app code.

Optimize your function app code: You can optimize your function app code to reduce the number of requests to the storage account and to minimize the amount of data being transferred. For example, you can use batch operations to upload multiple files at once, or you can compress the data before uploading it to reduce the amount of data being transferred.

Scale up the storage account: If the storage account is consistently experiencing high transaction volume and data ingress, you may need to scale up the storage account to handle the increased load. You can do this by upgrading the performance tier of the storage account or by increasing the number of storage account transactions allowed per second.

Use Azure CDN: You can use Azure CDN to cache frequently accessed data closer to your users, which can help reduce the load on the storage account and improve performance.
Malik, Deepankar 25 Reputation points

2024-04-16T12:13:32.72+00:00

@Sumarigo-MSFT

Below is the screenshot for the error messages

Transaction by storage type metric

We cannot increase the timeout value for the storage account as we need to return the response of the request to the client withing few milliseconds. Furthermore, we expect the load to increase on daily basics.

Would upgrading the performance tier to Premium help in my case?
Sumarigo-MSFT 43,801 Reputation points Microsoft Employee

2024-04-22T06:31:27.7033333+00:00

@Malik, Deepankar I would like to work closer on this issue! I've sent a private message requesting a few details. Could you please provide the necessary information in the private message?

Applications should retry requests that fails with an HTTP 503 or 500. Refer to the retry guidance here: https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#azure-storage
Throttling and ServerBusy errors here: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-performance-checklist#timeout-and-server-busy-errors

Azure premium storage: Design for high performance: https://learn.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance

1 answer

Vinodh247-1375 11,296 Reputation points

2024-04-16T07:32:30.4+00:00

Hi Malik, Deepankar,

Thanks for reaching out to Microsoft Q&A.

If your application is making a large number of requests in a short time, it could lead to throttling and the ADLS Gen2 service may throttle requests if there are too many read/write operations happening simultaneously. By default, the maximum request rate per storage account is 20,000 requests per second for standard Azure Storage accounts. For requests on a single blob, the limit is 500 requests per second. I am leaving a couple of links below which should help you identify and fix/optimize this issue.

Go through the below links and let me know if it didnt help.

https://learn.microsoft.com/en-us/answers/questions/1063199/adls-gen-2-errorcode-usererrordatastoreservicethro?orderBy=Newest

https://github.com/microsoft/AzureStorageExplorer/issues/3976

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment