Inconsistent performance issue in Azure Blob Storage Premium ZRS

Question

We are facing inconsistent performance issues repeatedly in Azure Blob Storage, particularly in the asynchronous copy blob process. Below is the code snippet we're using for copying:

 

We're initiating start copy requests that are queuing on the Azure storage without being processed. We aren't even receiving a server request ID to track the asynchronous operations, and without this ID, we're unable to cancel the requests that have already been initiated.

 

Sometimes, all requests are completed in milliseconds, while at other times, it takes more than 1 minute. Therefore, we included a workaround network timeout retry after 5 seconds. Now, most of the requests are taking 3-5 seconds and when retry it is getting completed in milliseconds.

 

Also, at that time, our service's CPU and memory usage are below 10%, so we don’t face any high load.

 

Provide a solution to resolve this performance issues.

Details:

 

Type	Details
Azure Storage account	Premium (Performance), Zone-redundant storage (Replication)
SDK	Azure.Storage.Blobs (12.15.0)

Inconsistent Behavior in Copy Blob:

Code used:

private static async Task CopyContainerAsync(
            BlobContainerClient sourceContainer,
            int pageSize,
            BlobContainerClient targetContainer,
            List<string> excludedBlobs = default)
        {
            var i = 0;
 
            await foreach (var batchItems in sourceContainer.GetBlobsAsync().AsPages(pageSizeHint: 100))
            {
                var batchTasks = new List<Task>();
                foreach (var item in batchItems.Values)
                {
                    if (excludedBlobs?.Contains(item.Name) == true)
                    {
                        continue;
                    }
 
                    BlobClient blob = sourceContainer.GetBlobClient(item.Name);
                    BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name);
                    batchTasks.Add(targetBlob.StartCopyFromUriAsync(blob.Uri));
                }
 
                await Task.WhenAll(batchTasks).ConfigureAwait(false);
            }
        }

BlobClient blob = sourceContainer.GetBlobClient(item.Name);
BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name);
await targetBlob.StartCopyFromUriAsync(blob.Uri, cancellationToken: cancellationToken)).configureawait(false);
 
 
services.AddAzureClients(builder =>
            {
                builder.AddBlobServiceClient(EnvironmentVariables.AzureBlobStorage)
                    .ConfigureOptions(options =>
                    {
                        options.Retry.MaxRetries = 5;
                        options.Retry.NetworkTimeout = TimeSpan.FromSeconds(5);
                    });
            });

Answer 1

Hi Muruga Anand - Thanks for reaching out over Q&A Forum.!

I would suggest you to start by enabling diagnostic logging and check whether the latency is server side one or E2E.

If the server side latency appears to be low then the troubleshooting need to be done outside of storage. I see you have reviewed some of the infra metrics which appears to be okay and in that case, you might need to explore for any networking blockers, packet drops, re-transmits etc happening on the network level.

If you observe high Server latency, in that case I would recommend raising a support ticket for deeper troubleshooting analysis.

Below are some of the reference links to review:

https://techcommunity.microsoft.com/t5/azure-paas-blog/how-to-isolate-latency-issue-for-azure-storage-account/ba-p/1430656

https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

Please let me know if there are any more queries and will be glad to assist.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.