How to safely copy large amount of files over the network using Powershell?

Question

Hi,

I have a script that copies very large amount of files from one windows server to another using powershell.

Right now I'm using this approach:

function makeSession() {
    $session = New-PSSession -Computer 1.1.1.1 -Credential $cred -UseSSL
    return $session
}

    $username = "user"
    $password = ConvertTo-SecureString -AsPlainText -Force -String "password"
    $cred = New-Object System.Management.Automation.PSCredential ($username, $password)
    $session = makeSession


Copy-Item –Path "C:\sourcefolder\courcefiles" –Destination "c:\destinationfolder\destinationfiles" -Force -Recurse –ToSession $session

Everything is fine, except SOMETIMES many files are still missing on the remote computer. For testing purposes, I have did launch this script with a single folder conaining 8000 files - this time all files has been copied successfully. So I think that there is a problem with either network integrity, of some files are being used when powershell is trying to copy them.

Searching for solution I came across resources where people are offering to use alternative methods, for example Start-BitsTransfer or robocopy. But unfortunately they are not designed to work with users not logged in to the network directly.

Any idea how to improve this script to be more reliable?

Thanks!!

Answer 1

Hi Lev

! Here's an example script that copies a large amount of files from one Windows server to another using PowerShell:

$sourcePath = "C:\Path\to\source\folder"
$destinationPath = "\\ServerName\C$\Path\to\destination\folder"

# Create a new instance of the Robocopy object
$robocopy = New-Object -TypeName System.Diagnostics.Process

# Set the Robocopy parameters
$robocopy.StartInfo.FileName = "robocopy.exe"
$robocopy.StartInfo.Arguments = "`"$sourcePath`" `"$destinationPath`" /E /ZB /MT:8 /R:3 /W:5 /NP /LOG+:C:\Path\to\log\file.log"

# Start the Robocopy process
$robocopy.Start() | Out-Null

# Wait for the Robocopy process to complete
$robocopy.WaitForExit()

# Check the exit code of the Robocopy process
if ($robocopy.ExitCode -eq 0) {
    Write-Host "File copy completed successfully."
} else {
    Write-Host "File copy failed. Exit code: $($robocopy.ExitCode)"
}

In this script, we use the robocopy command-line tool, which is built into Windows, to perform the file copy operation. The script sets various parameters for Robocopy, such as /E to include subdirectories, /ZB to use restartable mode (in case of interruptions), /MT:8 to use 8 threads for multithreaded copying, /R:3 to retry failed copies up to 3 times, /W:5 to wait 5 seconds between retries, /NP to exclude the progress percentage from the log, and /LOG+ to append the output to a log file.

You need to modify the $sourcePath, $destinationPath, and the /LOG+ argument to fit your specific paths and log file location. The Start() method starts the Robocopy process, and WaitForExit() waits for the process to complete. The script then checks the exit code of Robocopy to determine if the copy operation was successful.

Please make sure you have appropriate permissions and access rights on both the source and destination servers for the file copy to work correctly.

Answer 2

also, its strongly recommended that the files which saved in exceedingly long path should change to short path, it not copied

as example

\server\my old documentation for xxxx\ new folder\ xxxxxx\ folder.0\ files for 5.2023\xxxxxx\and then a very long file name

Answer 3

Here's an updated version of the script that includes providing credentials for network authentication when using Robocopy:

$sourcePath = "C:\Path\to\source\folder"
$destinationPath = "\\ServerName\C$\Path\to\destination\folder"

# Set the credentials for network authentication
$username = "Username"
$password = "Password"
$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$credentials = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $securePassword

# Create a new instance of the Robocopy object
$robocopy = New-Object -TypeName System.Diagnostics.Process

# Set the Robocopy parameters
$robocopy.StartInfo.FileName = "robocopy.exe"
$robocopy.StartInfo.Arguments = "`"$sourcePath`" `"$destinationPath`" /E /ZB /MT:8 /R:3 /W:5 /NP /LOG+:C:\Path\to\log\file.log"
$robocopy.StartInfo.UseShellExecute = $false
$robocopy.StartInfo.RedirectStandardOutput = $true
$robocopy.StartInfo.RedirectStandardError = $true
$robocopy.StartInfo.UserName = $credentials.UserName
$robocopy.StartInfo.Password = $credentials.Password

# Start the Robocopy process
$robocopy.Start() | Out-Null

# Wait for the Robocopy process to complete
$robocopy.WaitForExit()

# Check the exit code of the Robocopy process
if ($robocopy.ExitCode -eq 0) {
    Write-Host "File copy completed successfully."
} else {
    Write-Host "File copy failed. Exit code: $($robocopy.ExitCode)"
}

Make sure to replace "C:\Path\to\source\folder" with the actual path of the source folder and "\\ServerName\C$\Path\to\destination\folder" with the destination path on the remote server.

Set the correct Username and Password to provide the appropriate credentials for network authentication. The script converts the password to a secure string and creates a PSCredential object using the specified username and secure password.

The script creates a new instance of the System.Diagnostics.Process class to run Robocopy. It sets the necessary parameters for Robocopy, including the source and destination paths, and additional options such as /E for copying subdirectories, /ZB for using restartable mode, /MT:8 for multithreaded copying, /R:3 for the number of retries, /W:5 for the wait time between retries, /NP for excluding the progress percentage in the output, and /LOG+:C:\Path\to\log\file.log for logging the output to a file.

The script also sets the UseShellExecute, RedirectStandardOutput, RedirectStandardError, UserName, and Password properties of the StartInfo object to pass the credentials for network authentication to Robocopy.

After starting the Robocopy process, the script waits for it to complete and checks the exit code. If the exit code is 0, it displays a success message. Otherwise, it displays a failure message along with the exit code.

Please note that this script requires administrative privileges on both the local and remote computers to access the administrative share. Additionally, ensure that you have the necessary network permissions to copy files remotely using Robocopy.

Answer 4

Here's an updated version of your script that includes the credentials for network authentication:

function makeSession() {
    $session = New-PSSession -Computer 1.1.1.1 -Credential $cred -UseSSL
    return $session
}

$username = "user"
$password = ConvertTo-SecureString -AsPlainText -Force -String "password"
$cred = New-Object System.Management.Automation.PSCredential ($username, $password)
$session = makeSession

Copy-Item -Path "C:\sourcefolder\sourcefiles" -Destination "c:\destinationfolder\destinationfiles" -Force -Recurse -ToSession $session