What is the fastest way to send files to gen2 storage account?

Question

What is the fastest way to send files to gen2 storage account?

Kowalczyk Tomasz 0

Hey guys, I have a question regarding the fastest possible way to send files from folder to folder or storage account to storage account in general. I have around 100k small files, around 1kb each that needs to be send very quickly, ideally sub 1 minute time. What would be the best approach for this kind of thing? Cheers!

Kowalczyk Tomasz 0 Reputation points

2024-02-14T09:07:41.2933333+00:00

@KarishmaTiwari-MSFT hey, in terms of clarification, it could be both depending on which one is the fastes, I can send data from my own machine or from storage account to storage account, even from the same storage account from container to container in blob storage. The source is not the problem, the time kind of is. AzCopy sadly doesn't do the job, in storage explorer it take very long time to send over these files, up to an hour, the fastest way I checked was ADF using multiple concurrent copy activities around 1.5min for 100k files and was wondering if that time could be improved in any other way than adding more copy activities for lesser amount of files that would add up to 100k. thanks for the message!
KarishmaTiwari-MSFT 20,667 Reputation points Microsoft Employee

2024-02-16T04:46:43.7566667+00:00

@Kowalczyk Tomasz While you’ve already mentioned using ADF with multiple concurrent copy activities, it’s worth noting that ADF is a powerful tool for orchestrating data workflows. You can fine-tune your ADF pipeline by optimizing settings such as parallelism, batch size, and data flow transformations.

If you haven’t explored it yet, consider using ADF’s built-in copy activity specifically designed for Data Lake Storage Gen2. Ensure that you configure it appropriately for optimal performance 2.

Keep in mind that factors such as network latency, file size, and the number of files impact overall performance. For extremely large datasets, consider breaking them down into smaller chunks and uploading them in parallel.

Refer: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices
KarishmaTiwari-MSFT 20,667 Reputation points Microsoft Employee

2024-02-21T00:46:25.2566667+00:00

@Kowalczyk Tomasz Checking in to see if the comment above helped. Let me know. Thanks.

1 answer

Your answer

Kowalczyk Tomasz 0 Reputation points

2024-02-14T09:07:41.2933333+00:00

@KarishmaTiwari-MSFT hey, in terms of clarification, it could be both depending on which one is the fastes, I can send data from my own machine or from storage account to storage account, even from the same storage account from container to container in blob storage. The source is not the problem, the time kind of is. AzCopy sadly doesn't do the job, in storage explorer it take very long time to send over these files, up to an hour, the fastest way I checked was ADF using multiple concurrent copy activities around 1.5min for 100k files and was wondering if that time could be improved in any other way than adding more copy activities for lesser amount of files that would add up to 100k. thanks for the message!
KarishmaTiwari-MSFT 20,667 Reputation points Microsoft Employee

2024-02-16T04:46:43.7566667+00:00

@Kowalczyk Tomasz While you’ve already mentioned using ADF with multiple concurrent copy activities, it’s worth noting that ADF is a powerful tool for orchestrating data workflows. You can fine-tune your ADF pipeline by optimizing settings such as parallelism, batch size, and data flow transformations.

If you haven’t explored it yet, consider using ADF’s built-in copy activity specifically designed for Data Lake Storage Gen2. Ensure that you configure it appropriately for optimal performance 2.

Keep in mind that factors such as network latency, file size, and the number of files impact overall performance. For extremely large datasets, consider breaking them down into smaller chunks and uploading them in parallel.

Refer: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices
KarishmaTiwari-MSFT 20,667 Reputation points Microsoft Employee

2024-02-21T00:46:25.2566667+00:00

@Kowalczyk Tomasz Checking in to see if the comment above helped. Let me know. Thanks.

Answer 1

@Kowalczyk Tomasz Is your question about the fastest way to send files to Azure Data lake Gen2 storage account?
Can you please clarify on - from where are you sending the data from?

Please elaborate on your ask since it is not clear if you are planning to move data from one storage account to another or from a data source to Azure data lake storage account. Let me know and I can help you further.

When it comes to sending files to an Azure Data Lake Storage Gen2 account, you can use AzCopy (it’s a reliable choice for bulk data transfers), Azure Storage Explorer (if you prefer a graphical user interface) or Azure Data factory (if you need to automate data movement). However, for large-scale transfers, AzCopy might be more efficient.

While you’ve already mentioned using ADF with multiple concurrent copy activities, it’s worth noting that ADF is a powerful tool for orchestrating data workflows. You can fine-tune your ADF pipeline by optimizing settings such as parallelism, batch size, and data flow transformations.

If you haven’t explored it yet, consider using ADF’s built-in copy activity specifically designed for Data Lake Storage Gen2. Ensure that you configure it appropriately for optimal performance 2.

Keep in mind that factors such as network latency, file size, and the number of files impact overall performance. For extremely large datasets, consider breaking them down into smaller chunks and uploading them in parallel. Refer: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices

Share via

What is the fastest way to send files to gen2 storage account?

1 answer

Your answer