export files from sp document library

Tan 0 Reputation points
2024-10-01T02:29:03.6533333+00:00

Hi, I am trying to export files from sp document library to azure blob. I am using pnp core and trying to get files for past 3 years. since the files are in millions, so while fetching I am getting error as "The attempted operation is prohibited because it exceeds list view threshold". I am trying to fetch top 100 first then next 100 so that can get the files still it is giving error. Please suggest any other possible solution.

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,850 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,687 questions
{count} votes

2 answers

Sort by: Most helpful
  1. ChhaviKalra 155 Reputation points
    2024-10-01T05:23:17.66+00:00

  2. Nehruji R 7,801 Reputation points Microsoft Vendor
    2024-10-01T11:48:34.5633333+00:00

    Hello Tan,

    Greetings! Welcome to Microsoft Q&A Platform.

    I understand that you are receiving error "The attempted operation is prohibited because it exceeds list view threshold" when trying to fetch the files in SharePoint. This is because of the resourced throttling threshold value of 5000 set on SharePoint Online. The list view threshold ensures that users are not performing expensive operations on the SharePoint Online environment. Unfortunately, we can’t change this threshold value as we do in SharePoint On-Premises.

    You can store up to 30 million items or files in a list or library. Filtered views of large lists have a similar experience to other lists. However, when a list view shows more than 5000 items, you may run into a list view threshold error.

    In order to manage large lists and libraries do follow the below link,

    https://support.microsoft.com/en-us/office/manage-large-lists-and-libraries-b8588dae-9387-48c2-9248-c24122f07c59

    If not try to split your view into multiple views with the metadata filter conditions and ensure the query or filter returns less than 5k items.

    Steps:

    go to List settings >> Views >> Create view >> Standard view.

    User's image

    Set conditions in Filter section and ensure the filter returns the folder you want to copy and less than 5k items.

    You can also try using Microsoft Power Automate to copy files from SharePoint to Azure Blob Storage using a manual trigger, get file properties action, and Loop and create Blob actions.

    In Power Automate, we can create a cloud flow named like "copy files from SharePoint to Azure" and use the "Get files" action to select files from the SharePoint library. To move the files to Azure, we can use the "Create blob" action and establish an authentication between Power Automate and Azure using an access key. Please note the folder path and file name can be obtained from the dynamic properties of the "Get files" action, but the content needs to be obtained using an additional "Get file content" action.

    In order to move files from a SharePoint document library to an Azure Blob Storage Account using Power Automate, you must first identify the file identifier from the SharePoint site and add it to a loop in Power Automate. We can then move the action to read and move the file content inside the loop.

    Here is what the flow may look like: User's image

    User's image

    User's image

    Note: Power Automate has a maximum file size limit for individual files. This limit can vary based on the plan you are using, but typically, it's around 100 MB to 250 MB per file. In order to copy large files from SharePoint to Azure Blob Storage using Power Automate: https://sharepains.com/2022/11/15/copy-large-files-sharepoint-azure-blob/

    Similar queries for reference: is there any way to copy the entire folder as it is with all nested folders and files from sharepoint to adls - Microsoft Q&A

    https://learn.microsoft.com/en-us/answers/questions/1513111/how-to-copy-folders-and-files-from-a-sharepoint-si

    Do let us know if you have any further queries. I’m happy to assist you further.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.