export files from sp document library

Question

Hi, I am trying to export files from sp document library to azure blob. I am using pnp core and trying to get files for past 3 years. since the files are in thousands, so while fetching I am getting error as "The attempted operation is prohibited because it exceeds list view threshold". I am trying to fetch top 100 first then next 100 so that can get the files still it is giving error. Please suggest any other possible solution.

Answer 1

You could try using Azcopy command or Mover.

refer this.

https://learn.microsoft.com/en-us/answers/questions/1513111/how-to-copy-folders-and-files-from-a-sharepoint-si

Answer 2

Hello Tan,

Greetings! Welcome to Microsoft Q&A Platform.

I understand that you are receiving error "The attempted operation is prohibited because it exceeds list view threshold" when trying to fetch the files in SharePoint. This is because of the resourced throttling threshold value of 5000 set on SharePoint Online. The list view threshold ensures that users are not performing expensive operations on the SharePoint Online environment. Unfortunately, we can’t change this threshold value as we do in SharePoint On-Premises.

You can store up to 30 million items or files in a list or library. Filtered views of large lists have a similar experience to other lists. However, when a list view shows more than 5000 items, you may run into a list view threshold error.

In order to manage large lists and libraries do follow the below link,

https://support.microsoft.com/en-us/office/manage-large-lists-and-libraries-b8588dae-9387-48c2-9248-c24122f07c59

If not try to split your view into multiple views with the metadata filter conditions and ensure the query or filter returns less than 5k items.

Steps:

go to List settings >> Views >> Create view >> Standard view.

Set conditions in Filter section and ensure the filter returns the folder you want to copy and less than 5k items.

You can also try using Microsoft Power Automate to copy files from SharePoint to Azure Blob Storage using a manual trigger, get file properties action, and Loop and create Blob actions.

In Power Automate, we can create a cloud flow named like "copy files from SharePoint to Azure" and use the "Get files" action to select files from the SharePoint library. To move the files to Azure, we can use the "Create blob" action and establish an authentication between Power Automate and Azure using an access key. Please note the folder path and file name can be obtained from the dynamic properties of the "Get files" action, but the content needs to be obtained using an additional "Get file content" action.

In order to move files from a SharePoint document library to an Azure Blob Storage Account using Power Automate, you must first identify the file identifier from the SharePoint site and add it to a loop in Power Automate. We can then move the action to read and move the file content inside the loop.

Here is what the flow may look like:

Note: Power Automate has a maximum file size limit for individual files. This limit can vary based on the plan you are using, but typically, it's around 100 MB to 250 MB per file. In order to copy large files from SharePoint to Azure Blob Storage using Power Automate: https://sharepains.com/2022/11/15/copy-large-files-sharepoint-azure-blob/

Similar queries for reference: is there any way to copy the entire folder as it is with all nested folders and files from sharepoint to adls - Microsoft Q&A

https://learn.microsoft.com/en-us/answers/questions/1513111/how-to-copy-folders-and-files-from-a-sharepoint-si

Do let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.