Sharepoint Connector - ADF

Question

Hi,

I am analyzing sharepoint connector for my next project, we need to pull data from sharepoint and upload to azure storage account. It is going to be excel file with multiple sheets in sharepoint.

As part of analysis, I want to identify existing limitations/issues with sharepoint connector.

Any known issues and workarounds, and all other limitations.

Answer 1

Hi @Santhi Dhanuskodi

Thanks for the question and using MS Q&A platform.

Adding to the above answer, when working with SharePoint connectors to pull data and upload it to an Azure Storage Account, there are several known limitations and issues you might encounter. Here are some of the key points you should consider:

Authentication and Permissions:

• Issue: Authentication can be complex, especially if you're dealing with multiple environments (e.g., on-premises SharePoint vs. SharePoint Online).
• Workaround: Use Azure AD for authentication to simplify access and ensure proper permissions are configured.

File Size Limits:

• Issue: SharePoint has file size limits that can vary depending on the version (SharePoint Online vs. on-premises). The limit for SharePoint Online is 250 GB per file.
• Workaround: Ensure your Excel files are within the permissible limits or consider splitting larger files into smaller ones before uploading.

API Rate Limits:

• Issue: SharePoint API has throttling limits that can restrict the number of requests per minute/hour.
• Workaround: Implement exponential backoff and retry logic to handle throttling gracefully.

Metadata and Versioning:

• Issue: SharePoint maintains metadata and version history for files, which might not be directly transferable to Azure Storage.
• Workaround: If metadata is crucial, consider exporting it separately and storing it in a structured format (e.g., JSON) in Azure Storage.

Automation and Scheduling:

• Issue: Automating the data transfer process and scheduling it at regular intervals can be challenging.
• Workaround: Use Azure Logic Apps, Power Automate, or Azure Data Factory for scheduling and automation.

By keeping these limitations and workarounds in mind, you can better prepare for integrating SharePoint with Azure Storage and ensure a smoother data transfer process.

Below reference that might help you:

• You can also use Azure Data Factory
• https://www.youtube.com/watch?v=Nr2QJBbsBRU
• https://sharepains.com/2022/11/15/copy-large-files-sharepoint-azure-blob

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

The Sharepoint connector which we are discussing is the SPList connector and not the sharepoint library.

To copy data from sharepoint library to blob, you need to go the route of REST API as stated below :

https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/