copy data in a specific time daily

Question

I need a way to copy 6TB from windows server 2016 to windows server 2019 ( different place) with keeping all the permissions, I need this process to run daily at a specific time (10 pm to 5 am, then to be paused and resumed automatically at this specific time ) because we don't need this process to affect the daily company operation. then after the main operation is done (copying 6TB) the copy operation should be working daily at this specific time (10 pm to 5 am) to copy the changes only from the source to the destination.
Can anyone give me some suggestions?

• Blodriana

Answer 1

You can build a batch file, schedule it to start whenever, then force it to stop after the task has been running for x number of hours (an option within the task), then repeat the task each day starting at your specified time. each day check on the destination folder size until it has been completed.

Here are some options from searching.:

Windows has a built-in feature called DFS-R (Distributed File System Replication). The system syncs files between servers at some set interval.

There is a recommendation that I highly advise you to follow which is pre-seeding the target server with the initial data sync. Ideally, this would be done locally where you can 1) halt any file changes or do it at hours when users are not making changes to files, and 2) you're taking advantage of the local network's speed.

If you cannot move the target server to be in the same location as the source server, you can get a large external hard drive and use that to pre-seed the data using Robocopy. Then take the drive over to the other location to copy the data over. Robocopy is used as it will copy all file attributes.

One of the big things you need to be cognizant of is the issue of file locking. You can have a situation where if users at both ends are making changes to the file, the changes can cause file syncing/versioning control issues.

One of the nice things about DFS-R is it only copies the change blocks for any data that changes. So it provides some efficiency in replicating data across limited bandwidth WAN/Internet connections.

Best practices would be to have a VPN tunnel established between the two sites.

Scheduling-wise, you will have to look into the granularity of the scheduling DFS-R offers.

Other considerations.....your experience will be totally dependent on the change rate and what files do get changed/added. For video files, these are the worst to replicate as these files don't compress or dedupe well. The DFS-R database has to be synced between the two servers so be aware of this. In some situations, the file syncs are never complete due to the sheer volume of the number of new/changed files having to be indexed where the syncing of the database doesn't complete or consumes the replication window before the file transfers begin or can be completed.

Also, there are som straightforward backup tools that can do these requirements with just a few steps automatically, take a look at GoodSync, Gs Richcopy 360, and AllwaySync,

Answer 2

I believe you could use PowerShell script and use Copy-item to copy items and create script and define to perform this task.
Then you may use Task Scheduler and define the date and time which task should be run or set to repeat every day and instruct it to run the screen you have written earlier.
I start by running task for few files and in limit time first to make sure everything works as expected and then do it for the entire items.

Answer 3

copy the changes only from the source to the destination

Use robocopy with the /mir and /sec switches.

https://social.technet.microsoft.com/wiki/contents/articles/1073.robocopy-and-a-few-examples.aspx

The first run will take a while because the entire 6tb needs to be copied. But subsequent runs will be faster because robocopy will only copy changed files. You will need to test to see how long those take. You may not need to kill the robocopy process.

You can also throttle the amount of data that is copied.

::
:: Copy File Throttling Options :
::
/IoMaxSize:n[KMG] :: Requested max i/o size per {read,write} cycle, in n [KMG] bytes.
/IoRate:n[KMG] :: Requested i/o rate, in n [KMG] bytes per second.
/Threshold:n[KMG] :: File size threshold for throttling, in n [KMG] bytes (see Remarks).

One important item: all security permissions should be done with groups, and not individual user accounts. Adding a user to a group to grant them access to a folder will not be seen as a change to the file system. If you add a user to the NFS permissions on the root folder, then would be seen as a change and robocopy would recopy all 6tb of data. (Disclaimer: that is how it worked the last time that I tested. I do not know if the latest version of robocopy still acts that way.)

Run robocopy interactively to see how it works and monitor performance. Then create a schedule task to fully automate the process.

Answer 4

@Blodriana James

Everything you describe can be handled using DFS-R. However, given the size it is highly recommended to pre-seed the data prior to the configuration.

The reason why I think DFS-R seems like a good match is the following

1. It will copy data between servers.
2. It will only copy the changed files
3. It will keep permissions on the file level intact
4. You can create a schedule as to when it should run (from 10 PM - 5 am)
5. You have a log of the file move
6. You are not using a custom script. So if something stops working you can call for help by a lot of communities (including this one).
7. Optional: you can create a namespace for the server. So if you have a domain (let's call it comodo.com here), then you can create a space named comodo.com\files that will point to the server your wish people to work on (or both if you want load balancing). If you for some reason wan't all client to use the other server, then all you have to do is change the server in DFS-R, and traffic will be sent to the new server.
8. It's free and part of your server license.

There are lots of material describing how it works, but from your description here, that would be my suggestion for you.

However, this would only work if you have a domain (which i assume you have). If you have multiple domain controllers today, then you already use it since servers from 2012R2 and ahead uses DFS-r to replicate sysvol :)

You will find plenty of material on how to configure it step-by-step like one of the following and many more.

https://www.vembu.com/blog/distributed-file-system-dfs-windows-server-2016-brief-overview/
configure-dfs-replication-for-windows-server

And else, there are many inhere who can help you :)

One last detail: since this is something that from your description should run continuously and there is a service that does the same. I would not recommend you to create a script for it.

Answer 5

first I need to know, what are these files? the network bandwidth ? have any storage like NAS? the rate of the modified data?