Very slow when writing lots of files to Blob Storage

Question

I wrote about 1000 small files(~0.5 mb/file) to blob storage from ADF.
I only found 70 files by Azure sotrage explorer after ADF activity finished.

When I refresh, there will be 1 or 2 files added to the folder like the following image show.

My question is, are all files wrote to blob storage at that time when ADF activity finsihed. Because I have another activity to consume these files but I don't know when all the files can be writen to blob storage?

Answer 1

anonymous user-0048 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

There could be several reason, for the performance issue (Network bandwidth, intermittently issue may occur). Can you perform
https://www.azurespeed.com/Azure/Download

Which version on Azure Storage explorer are using? Can you try to un-install and re-install the latest version of Storage Explorer.

For testing purpose you can also use azcopy tool and check the performance

Storage Explorer currently doesn't have the functionality for users to configure our usage of the network bandwidth. It is something we want to enable in the future as a part of the configuration. There are two workarounds I can think of in addition to using FTP. Storage Explorer has a built-in Experimental feature called AzCopy integration, which dramatically speeds up blob uploading and downloading. You can also use AzCopy at command line which actually gives you the ability to configure different parameters including the bandwidth you want to use during upload/download. Hope this is informational.

Additional information: There is no throttling on the Azure side below the published 60 MB/s scalability target for a single blob. If the network and client machine can handle the traffic then we will send it.

Here's a guide for troubleshooting high E2E latency: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting#metrics-show-high-AverageE2ELatency-and-low-AverageServerLatency

Kindly let us know if the above helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.