Share via

How to move a large file library (4M+ files, 3+TB) from Onedrive for Business to an external drive

Anonymous
2023-11-03T11:54:07+00:00

I have a problem with Onedrive. My university provides a large Onedrive cloud storage (located at the sharepoint.com domain), so I had been using it to backup data. Over the years, I've accumulated terabytes of data in millions of files. Now I want to get the data out of Onedrive to an external HDD. I've found that everywhere here on the forums it is recommended to use the onedrive app (File Explorer) to do that. However, now I've set up a "test move" of about 2000 files (a little less than 10 GB) and Onedrive has been frozen (Not responding) for over 30 minutes, only 2 files (24kB total) have been moved to the external drive in that time - I did it just like it was recommended in many threads here already: Open Onedrive folder with cloud icons next to files, select a folder and drag it to the external drive, the sync icon in windows toolbar says every 10-15 seconds it synced one file, likely downloading it to local storage (I cannot check, File Explorer stops responding when I try to open the Onedrive folder).

I've restarted Onedrive (killed it and launched it again) multiple times now, this is a third attempt at the 10 GB move. How am I supposed to transfer TBs of files when just 10 GB clogs the system like this?

My internet connection is 1Gbit/s, my download speed from other sites is around 100 MB/s.

I have only about 500 GB free space on my local computer, so I'd rather avoid moving the files to my local PC and then transferring it to the external HDD (some of my Onedrive folders ale larger than 500 GB). Is there something like a Sharepoint Migration Tool that would work the other way around, for downloads, or any way to move large number of files away from Onedrive?

Microsoft 365 and Office | OneDrive | For business | Windows

Locked Question. This question was migrated from the Microsoft Support Community. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Anonymous
    2023-11-08T11:13:53+00:00

    So, after a few days of searching and trying various approaches, I've found out the following:

    No matter what client I use (default Onedrive client, RClone/command line/Powershell, setting up a custom API through Azure portal...), it will not run faster. Onedrive itself enforces a limit for calls/requests to its servers, there is no way around it. If you hit the "transactions" limit, Onedrive will slow down/throttle your connection automatically. Unfortunately, it is a restriction "per request", not "per bytes transferred" (as far as I've noticed), so if you have lots of small files that are on Onedrive individually, good luck retrieving them. In this case, doing a one-off backup of a large archive should work fine. However, this has caveats as well.

    Just to sum up the experience: Onedrive client (File Explorer) seems to natively limit the number of transactions it processes so that you don't hit this limit where you'd have to wait several minutes before being allowed to contact the Onedrive server again. RClone/Powershell/cmd commands don't have this restrictions and will hit this limit, so while you may be able to batch-transfer several hundreds of files in seconds, you'll then spend some minutes waiting before being allowed to continue, essentially reaching the same average speed as the regular Onedrive client.

    While Sharepoint Migration Tool was able to upload large numbers of files quickly, it was due to the tool packaging the files into binary chunks of larger size and uploading them in that bulk, unpacking them on Onedrive. However, Onedrive stopped supporting zipping some time ago, so there is no way to zip files on Onedrive - you can request a folder download, which will automatically zip the folder you want, but it will zip it on the fly, meaning you still hit the enforced "requests per second" limit, as each file processed by Onedrive will count towards this limit. This is also very inconveniet if you decide to upload your backups as archives - if you want to change one specific file, you'll have to download and reupload the whole archive, as Onedrive stopped supporting archive operations on the server. I vaguely remember I used to be able to zip/unzip files in the web interface, but I might be wrong, it might have been another cloud service, so maybe Onedrive never supported it.

    It seems to be a very much business decision (Onedrive will zip the data for you automatically when you want to download it, so it is not like it would not be able to work with archives), you are provided with a tool that can upload lots of data (so OneDrive is capable of operating at high speeds), but when you want to retrieve the data and/or move them elsewhere you just have to brace yourself and grit your teeth (because the speeds available to the users are artificially reduced).

    Maybe this will be helpful to other people facing the same issue.

    20+ people found this answer helpful.
    0 comments
  2. Anonymous
    2023-11-04T06:19:53+00:00

    Dear Tomas Malina,

    Thanks for writing back to us.

    As per the further updates, to my experience I would not recommend you to move the data in your OneDrive for Business to the external drive with the help of the OneDrive sync client because it will take a lot of time and will cause the sync issue due to the many files (where I recommend either using Power Automate or custom PowerShell script as mentioned in my previous reply).

    If you are comfortable with the ZIP files, though the OneDrive for Business can't convert the group of the files into the ZIP files but when you either download the folder or download the multiple files from OneDrive for Business, it will be downloaded as a ZIP file, see the below screenshot:

    .

    So, you can make different folders in your OneDrive for Business (in browser) and move the content into the folders and files into the folders created. Then download the folders batch wise and move them to your external harddrive.

    Appreciate your patience with us and have a great weekend!!

    Best Regards,

    Sophia

    0 comments
  3. Anonymous
    2023-11-03T16:14:15+00:00

    Hi, thanks for your reply.

    I have just tried the approach to download the files to my local PC in batches (selecting them to be always kept locally), but the synchronization is painfully slow - if the file is small (KBs in size), it can process only about 50 files per minute, if the file is large, it will get capped by my Gbit connection (which is fine). However, I have millions of files on OneDrive, just a quick calculation gives me that if I let the sync run constantly, it will take months (!!) to recover my files from OneDrive, which is unacceptable for me. On Google drive, for example, I can download large numbers of files from a browser by packing it into a zip archive (I've tested it with several hundreds of GB worth of files and it worked), is there for example a way to package my library on Onedrive and download it in batches like this..? I can extract them later. Or any other way that will not take months..?

    Edit: I've moved some large files (a few thousand files, approx 500 GB), but I am now facing the problem to move the small files, which it seems to clog up on...

    0 comments
  4. Anonymous
    2023-11-03T13:24:08+00:00

    Dear Tomas Malina,

    Good day!!

    I understand your concern and to my knowledge, even OneDrive sync client will help you to move the data from the OneDrive for Business to your external drive, it will take lot of time because of the number of files and storage you are having your OneDrive, where you need to take the steps on batch wise such as splitting your data into different folders and then sync only specific folder from OneDrive for Business and then copy the data to the external hard drive. After pasting them to external drive successfully, unlinking the synced folder and following the same process for remaining folders.

    Moreover, SharePoint Migration Tool will not be helpful because it moves the data to SharePoint Online. So, in this situation, you can try for Power Automate or Custom PowerShell script.

    For Power Automate, I suggest you to give a shot by posting your concern in the Microsoft Power Automate Community - Power Platform Community to get the detailed help from the experts.

    For custom PowerShell script, I suggest you to post your concern in the related Microsoft community i.e., Questions - Microsoft Q&A to get the detailed help from the experts.

    In addition, I will also keep this thread open so that experts and members who faced related situation in past will share their opinion on your requirement.

    Appreciate your patience and understanding. Have a great day!!

    Best Regards,

    Sophia

    0 comments