How to upload 20 Million documents to sharepoint using graph api & c#

Question

I have requirement to upload 20 Million documents from one of legacy application to SharePoint. When I use Graph API using C#, I am able to upload 9-10 documents per min which is very low.

What are the ways to upload 20 Million documents to SharePoint without throttling?

Answer 1

• Yes, Sharepoint has throttling mechanisms to protect the service. When you exceed usage limits, SharePoint Online might slow down or stop your processes.

This is likely the reason you're only able to upload 9-10 documents per minute.

• Try optimizing your code by using BAtch requests in Microsoft Graph, This allows you to send multiple requests in a single HTTP call, which can significantly reduce the time it takes to upload documents. Also, worth considering is the use of parallel processing within C#.

Let me know if you need a sample code for this

How to Recognize Throttling?

When you're being throttled, SharePoint Online will return a 429 Too Many Requests HTTP status code. Additionally, a Retry-After HTTP header will often be included to indicate how many seconds you should wait before making another request.

How to Minimize the Risk of Being Throttled?

Use Exponential Backoff: When you get throttled, don't immediately retry the request. Instead, implement an exponential backoff strategy. This means you wait for a short duration after the first failure, and if the request fails again, you double the wait time, and so on.

Optimize Your Queries: Ensure your requests to SharePoint are as efficient as possible. For example, when querying lists, only retrieve the fields you need rather than all fields.

Answer 2

Hi @Priyanka Vaddapelli,

Per my research, I do agree with Brian Zarb's answer. Since there is a throttling limit in SharePoint, I will recommend you to use Migration Manager. Here is a similar issue with yours, please make a reference.

https://learn.microsoft.com/en-us/answers/questions/778011/uploading-huge-amounts-of-data-to-sharepoint

---

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.