How to copy folders and files from a SharePoint site to Azure blob storage?

Question

How to copy folders and files from a SharePoint site to Azure blob storage? I've already tried using Power Automate and Logic Apps but there is a file size limitation, for example 100MB. I've also tried using Azure Data Factory using the Copy component, but I can only save in binary format. Is there an easier solution to copying folders and files? I'm talking about a large volume of data on the sharepoint website. Thanks!

Answer 1

Hello Ítalo Magalhães da Silva,

Thank you for posting your query here!

To transfer large files from SharePoint to Azure Blob Storage, there are several methods you can consider, as outlined in Microsoft's documentation:

You can use Microsoft Power Automate: You can use Power Automate for copying files from a SharePoint folder to an Azure Blob folder, though this might have limitations. Power Automate has a maximum file size limit for individual files. This limit can vary based on the plan you are using, but typically, it's around 100 MB to 250 MB per file.

You can also use Azure Data Factory: It is a cloud-based data integration service that allows you to create data pipelines that move and transform data between various sources and destinations, including SharePoint and Azure Blob Storage. You can create a pipeline that retrieves files from SharePoint using the SharePoint connector and then uses the Azure Blob Storage connector to upload the files to Azure Blob Storage. This approach requires some configuration and coding but provides more flexibility and scalability than Logic Apps.

As you mentioned, you might run into some issues with large files and Logic Apps. If there are extremely large files to be copied from that SharePoint library. SharePoint has a default limit of 100 MB buffer size, and the Get File Content action doesn’t natively support chunking.

You can refer this article to use a binary dataset if you just want to copy the full file rather than read the data. https://datasavvy.me/2021/12/07/copying-large-files-from-sharepoint-online/

Other reference that might help you: https://sharepains.com/2022/11/15/copy-large-files-sharepoint-azure-blob/

Also check Azure Data Box: It is a physical device provided by Microsoft for offline data transfer. This method is suitable for large data sets and can significantly reduce the time it takes to transfer the data.

You may also consider using third-party tools like Gs Richcopy 360, ShareGate, and GoodSync.

Note: The third-party articles are recommended to help you find alternatives, Microsoft does not verify if the third-party workarounds work completely.

Do let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Consider using the AzCopy CLI Tool:

• Powerful command-line tool for transferring data to and from Azure storage.
• No size limitations with the --include parameter.
• Supports parallel transfers and multi-threading for speed optimization.
• Requires scripting knowledge and command-line comfort.

Or Consider using Third-party migration tools:

• Consider tools like AvePoint, Gs Richcopy360, or ShareGate.
• Often offer intuitive interfaces, automated workflows, and advanced features.
• Might have licensing costs or limitations in their free versions.

These should help copy data from a SharePoint site to Azure blob directly