Download 10 Million files from SharePoint Online - C#

Question

Download 10 Million files from SharePoint Online - C#

Amit Singh Rawat 731

I want to download and process files from my SharePoint Online environment. For now ignore Processing Files. I have around 10Millions files in my SP Online Environment. Can anyone suggest a way to download all these files in a fastest way possible.

I want to use C# with .NET6 and above. I have all SP admin, Global Admin Privileges.

Thank you in advance.

Brian Zarb 1,670

HI Amit, of course you can do that but it's quite a hefty task. I recently downloaded a large number (not 10 million) but here is how i managed to do it:

There are a couple of things to remember before we start. We have to use things such as Parallel processing as this will increase performance (sacrificed your computers resources).
Sharepoint Throttling is also something else that you have to consider as it might limit you on how many files you can download at once. Here is more about that: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

okay so lets get started:

Set Up the SharePoint Client Library

Add the SharePointPnPCoreOnline NuGet package to your project.

Create a Configuration to Store SharePoint Site URL, Credentials, and Download Path

public class SharePointConfig
{
    public string SiteUrl { get; set; }
    public string Username { get; set; }
    public string Password { get; set; }
    public string DownloadFolder { get; set; }
}

Create a Class for SharePoint Operations

public class SharePointService
{
    private readonly SharePointConfig _config;
    public SharePointService(SharePointConfig config)
    {
        _config = config;
    }
    public void DownloadFiles(string libraryName)
    {
        using (var ctx = new ClientContext(_config.SiteUrl))
        {
            SecureString password = new SecureString();
            foreach (char c in _config.Password.ToCharArray()) password.AppendChar(c);
            ctx.Credentials = new SharePointOnlineCredentials(_config.Username, password);
            // Get the SharePoint library
            var list = ctx.Web.Lists.GetByTitle(libraryName);
            var query = CamlQuery.CreateAllItemsQuery();
            var items = list.GetItems(query);
            ctx.Load(items, its => its.Include(item => item.File));
            ctx.ExecuteQuery();
            // Download files
            Parallel.ForEach(items, new ParallelOptions { MaxDegreeOfParallelism = 8 }, item => // You can adjust parallelism level based on your system capability
            {
                var fileRef = item.File.ServerRelativeUrl;
                var fileInfo = File.OpenBinaryDirect(ctx, fileRef);
                var filePath = $"{_config.DownloadFolder}/{Path.GetFileName(fileRef)}";
                using (var fileStream = System.IO.File.Create(filePath))
                {
                    fileInfo.Stream.CopyTo(fileStream);
                }
            });
        }
    }
}

Use the Above Service in Your Main Code

Adjust the below code based on your own parameters

var config = new SharePointConfig
{
    SiteUrl = "<Your SharePoint Site URL>",
    Username = "<Your Username>",
    Password = "<Your Password>",
    DownloadFolder = "<Download Path>"
};
var service = new SharePointService(config);
service.DownloadFiles("<Your Document Library Name>");

Please test this with some small repo before you go live on the actual 10 mil., if you found this helpful please mark it as your answer. Thank you

Share via

Download 10 Million files from SharePoint Online - C#

Your answer