Best Practices for Using Robocopy with Large Data Sets

Question

Hi,

In a tech refresh project, migration of 5–10 years of data using Robocopy is planned. Each month's data consists of approximately 2–3 GB, containing a substantial number of files and folders. What are the practical or documented limits for the number of files and total data size that Robocopy can handle reliably? Additionally, what best practices or system resource considerations should be taken into account for successful large-scale transfers?

Answer 1

The main issue will likely be the network throughput. If your source and destination servers are on the same subnet, then robocopy should perform well. If the path between your servers needs to traverse firewalls and multiple routers, then the time to do the migration will take longer. You would need to run some tests to measure the performance of your environment.

You would also want to run robocopy.exe on one of the servers and not from a desktop. Running from a desktop would incur additional, unnecessary network traffic.

Another consideration is the I/O activity and performance on the disk subsystem on the source and destination servers. If the current workload on your servers is I/O intensive, then robocopy is just going to add to it. You may need to run robocopy off hours when activity is lower.

You can also adjust the /MT parameter and utilize robocopy's throttling options if necessary. From robocopy /help.

	/MT[:n] :: Do multi-threaded copies with n threads (default 8).
			   n must be at least 1 and not greater than 128.
			   This option is incompatible with the /IPG and /EFSRAW options.

	   Redirect output using /LOG option for better performance.
       If either /IoMaxSize or /IoRate are specified, robocopy will enable
       copy file throttling (the purpose being to reduce system load).
       Both may be adjusted to allowable or optimal values; i.e., both
       specify desired copy parameters, but the system and robocopy are
       allowed to adjust them to reasonable/allowed values as necessary.
       If /Threshold is also used, it specifies a minimum file size for
       engaging throttling; files below that size will not be throttled.
       Values for all three parameters may be followed by an optional suffix
       character from the set [KMG] (kilo, mega, giga).

For the migrations that I have done in the past, we used the /MIR switch and run robocopy at least twice. The first run was to copy over the bulk of the files. Subsequent runs will only copy files that have changed or been deleted. To MIRror the files/folders.

I don't recall ever having to tweak the /MT or throttle switches, we just let it run. Depending on the server we might even run it a second time depending on how many files were updated and if we had any delays in the migration timeline. We also used the /R:0 switch to get robocopy to skip over any files that might be in use.

On the destination server we would disable our antivirus software (McAfee) because we had it configured to scan on file write, and there was no need to do an AV scan during the migration. That would just slow down the process.

When it came time to do the final cutover, we would stop any applications and disable the shares on the source server. That would prevent any updates to the source files. We would run robocopy without the /R:0 switch so that we didn't skip over any files.

I would suggest that you pick a subfolder and run a test to see how long that copy takes. That will give you an idea of how long a given number of files takes to be copied and if it impacts performance to the point where you have to tell robocopy to throttle the copy process.

I have also found it useful to test with the /l switch too to see what files would be copied.