Inconsistent performance issue in Azure Blob Storage Premium ZRS

Muruga Anand 26 Reputation points
2024-05-02T05:37:22.67+00:00

We are facing inconsistent performance issues repeatedly in Azure Blob Storage, particularly in the asynchronous copy blob process. Below is the code snippet we're using for copying:

 

We're initiating start copy requests that are queuing on the Azure storage without being processed. We aren't even receiving a server request ID to track the asynchronous operations, and without this ID, we're unable to cancel the requests that have already been initiated.

 

Sometimes, all requests are completed in milliseconds, while at other times, it takes more than 1 minute. Therefore, we included a workaround network timeout retry after 5 seconds. Now, most of the requests are taking 3-5 seconds and when retry it is getting completed in milliseconds.

 

Also, at that time, our service's CPU and memory usage are below 10%, so we don’t face any high load.

 

Provide a solution to resolve this performance issues.

Details:

 

Type Details
Azure Storage account Premium (Performance), Zone-redundant storage (Replication)
SDK Azure.Storage.Blobs (12.15.0)

Inconsistent Behavior in Copy Blob:

Aspose.Words.e9ebf2b5-c5c0-4f9c-a280-f17e0800938f.001

Aspose.Words.e9ebf2b5-c5c0-4f9c-a280-f17e0800938f.002

Code used:

private static async Task CopyContainerAsync(
            BlobContainerClient sourceContainer,
            int pageSize,
            BlobContainerClient targetContainer,
            List<string> excludedBlobs = default)
        {
            var i = 0;
 
            await foreach (var batchItems in sourceContainer.GetBlobsAsync().AsPages(pageSizeHint: 100))
            {
                var batchTasks = new List<Task>();
                foreach (var item in batchItems.Values)
                {
                    if (excludedBlobs?.Contains(item.Name) == true)
                    {
                        continue;
                    }
 
                    BlobClient blob = sourceContainer.GetBlobClient(item.Name);
                    BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name);
                    batchTasks.Add(targetBlob.StartCopyFromUriAsync(blob.Uri));
                }
 
                await Task.WhenAll(batchTasks).ConfigureAwait(false);
            }
        }
BlobClient blob = sourceContainer.GetBlobClient(item.Name);
BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name);
await targetBlob.StartCopyFromUriAsync(blob.Uri, cancellationToken: cancellationToken)).configureawait(false);
 
 
services.AddAzureClients(builder =>
            {
                builder.AddBlobServiceClient(EnvironmentVariables.AzureBlobStorage)
                    .ConfigureOptions(options =>
                    {
                        options.Retry.MaxRetries = 5;
                        options.Retry.NetworkTimeout = TimeSpan.FromSeconds(5);
                    });
            });
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,769 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,493 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Amrinder Singh 3,260 Reputation points Microsoft Employee
    2024-05-02T06:30:51.6966667+00:00

    Hi Muruga Anand - Thanks for reaching out over Q&A Forum.!

    I would suggest you to start by enabling diagnostic logging and check whether the latency is server side one or E2E.

    If the server side latency appears to be low then the troubleshooting need to be done outside of storage. I see you have reviewed some of the infra metrics which appears to be okay and in that case, you might need to explore for any networking blockers, packet drops, re-transmits etc happening on the network level.

    If you observe high Server latency, in that case I would recommend raising a support ticket for deeper troubleshooting analysis.

    Below are some of the reference links to review:

    https://techcommunity.microsoft.com/t5/azure-paas-blog/how-to-isolate-latency-issue-for-azure-storage-account/ba-p/1430656

    https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

    Please let me know if there are any more queries and will be glad to assist.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments