Faster Bulk filecopy to remote server using powershell

Question

I have a situation where I have to setup a normal windows file copy operation to a group of remote servers behind a loadbalancer using powershell.

This is what I have already where I am passing a bunch of parameters and able to perform file copy on one of the servers. And currently the copy operation takes almost 25 mins to copy 1GB of data which is what I want your help in optimizing the script in a better way. And the number of servers behind the LB can go up to 5 servers and if the script is not optimized it will take very long to complete the entire copy operation. As far as getting the list of active servers from the AWS ELB, I can take care of that part. But please suggest how the time taken to copy the files can be reduced. At present I am using copy-item to perform the copy operation and here is my complete script so far

param (
[string]$targetGroups,
[string]$adminUserName,
[string]$adminPassword,
[string]$resourceFilteringMethod,
[string]$sourcePath,
[string]$targetPath,
[string]$additionalArguments,
[string]$cleanTargetBeforeCopy,
[string]$accessKey,
[string]$secretKey,
[string]$drainConnections
)

Import-Module "AWSPowerShell"
[bool]$removeFromPool = "Convert-String $drainConnections Boolean"
$sourcePath = $sourcePath.Trim('"')
$targetPath = $targetPath.Trim('"')
$adminPassword = $adminPassword.Trim('"')
$adminUserName = $adminUserName.Trim('"')

[ValidateNotNullOrEmpty()] $userPassword = "$adminPassword"
$userPassword = ConvertTo-SecureString -String $userPassword -AsPlainText -Force
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $adminUserName, $userPassword

$session = New-PSSession -ComputerName $targetGroups -Credential $userCredential

Copy-Item -Path $sourcePath -Destination $targetPath -Recurse -Verbose -ToSession $session -ErrorAction SilentlyContinue -Force

Please suggest a better way to perform bulk copy faster

Answer 1

You could try using BITS to do the transfer: bitstransfer

Answer 2

Also I tried this script and I get the below error

script:
Import-Module BitsTransfer
$Source="C:\builds\builddir\artifact"
$Destination="\dest-servername\d$\dest\folder"
if ( -Not (Test-Path $Destination))
{
$null = New-Item -Path $Destination -ItemType Directory
}
$folders = Get-ChildItem -Name -Path $source -Directory -Recurse
[ValidateNotNullOrEmpty()] $userPassword = "$password"
$userPassword = ConvertTo-SecureString -String $userPassword -AsPlainText -Force
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $userPassword
$session = New-PSSession -ComputerName dest-servername -Credential $userCredential
$bitsjob = Start-BitsTransfer -Source $Source*.* -Destination $Destination -asynchronous -Priority low
while( ($bitsjob.JobState.ToString() -eq 'Transferring') -or ($bitsjob.JobState.ToString() -eq 'Connecting') )
{
Sleep 4
}
Complete-BitsTransfer -BitsJob $bitsjob
foreach ($i in $folders)
{
$exists = Test-Path $Destination\$i
if ($exists -eq $false) {New-Item $Destination\$i -ItemType Directory}
$bitsjob = Start-BitsTransfer -Source $Source\$i*.* -Destination $Destination\$i -asynchronous -Priority low
while( ($bitsjob.JobState.ToString() -eq 'Transferring') -or ($bitsjob.JobState.ToString() -eq 'Connecting') )
{
Sleep 4
}
Complete-BitsTransfer -BitsJob $bitsjob
}

When I run this script I get the below error

You cannot call a method on a null-valued expression.
At bits-transfer.ps1:14 char:8

• while( ($bitsjob.JobState.ToString() -eq 'Transferring') -or ($bitsjo ...
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• CategoryInfo : InvalidOperation: (:) [], RuntimeException
• FullyQualifiedErrorId : InvokeMethodOnNull

Complete-BitsTransfer : Cannot validate argument on parameter 'BitsJob'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
At bits-transfer.ps1:18 char:32

• Complete-BitsTransfer -BitsJob $bitsjob
• ~~~~~~~~
• CategoryInfo : InvalidData: (:) [Complete-BitsTransfer], ParameterBindingValidationException
• FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.BackgroundIntelligentTransfer.Management.CompleteBitsTransferCommand

@Rich Matheisen any suggestion?