I want to bulk insert from blob-to-blob. where I am having files names text files

Question

I want to bulk insert from blob-to-blob. where I am having files names text files

function Copy-AzureBlobWithFileList {
    param (
        [string]$SourcePath,     # e.g. https://account.blob.core.windows.net/container/folder?SAS
        [string]$DestPath,       # e.g. https://account.blob.core.windows.net/container2/folder?SAS
        [string]$FileListPath, # e.g. "CR121.PDF;CR234.PDF" or path to a file
        [string]$PeriodYear      # just used for naming the include file
    )

    $includeFilePath = "D:\CRS1-841\StagingFolder\$PeriodYear.txt"
    $folder = Split-Path $includeFilePath

    if (-not (Test-Path $folder)) {
        New-Item -ItemType Directory -Path $folder | Out-Null
    }

    # Decide if $FileListString is a file path or semicolon separated list
    if (Test-Path $FileListPath) {
        # It's a path to a file
        $fileNames = Get-Content -Path $FileListPath | ForEach-Object { $_.Trim() }
    } else {
        # It's a string with filenames separated by ;
        $fileNames = $FileListPath -split ';' | ForEach-Object { $_.Trim() }
    }

    # Write the filtered list to include file
    $fileNames | Set-Content -Encoding UTF8 -Path $includeFilePath

    $args = @(
        "copy",
        $SourcePath,
        $DestPath,
        "--include-path=$includeFilePath",
        "--recursive=true",
        "--overwrite=ifSourceNewer"
    )

    Write-Output "Running azcopy $($args -join ' ')"
    Start-Process -FilePath "azcopy" -ArgumentList $args -NoNewWindow -Wait
}

Sampath 3,750 Microsoft External Staff Moderator

Hello @R, Franklin (AssetMark, Contractor) ,
Steps to Bulk Copy Files from Blob to Blob Using AzCopy

Download and install AzCopy from Microsoft Docs.
Your script is mostly correct, but ensure that the include-path parameter is properly formatted.

Example:


     function Copy-AzureBlobWithFileList {

         param (

             [string]$SourcePath,     # Source Blob URL with SAS token

             [string]$DestPath,       # Destination Blob URL with SAS token

             [string]$FileListPath,   # File list (semicolon-separated or file path)

             [string]$PeriodYear      # Naming reference

         )

         $includeFilePath = "D:\CRS1-841\StagingFolder\$PeriodYear.txt"

         $folder = Split-Path $includeFilePath

         if (-not (Test-Path $folder)) {

             New-Item -ItemType Directory -Path $folder | Out-Null

         }

         # Read file names from input

         if (Test-Path $FileListPath) {

             $fileNames = Get-Content -Path $FileListPath | ForEach-Object { $_.Trim() }

         } else {

             $fileNames = $FileListPath -split ';' | ForEach-Object { $_.Trim() }

         }

         # Write file names to include file

         $fileNames | Set-Content -Encoding UTF8 -Path $includeFilePath

         # Run AzCopy command

         $args = @(

             "copy",

             $SourcePath,

             $DestPath,

             "--include-pattern=$($fileNames -join ';')",

             "--recursive=true",

             "--overwrite=ifSourceNewer"

         )

         Write-Output "Running azcopy $($args -join ' ')"

         Start-Process -FilePath "azcopy" -ArgumentList $args -NoNewWindow -Wait

     }

Ensure that both SourcePath and DestPath have SAS tokens with read and write permissions.

Before running the script, test AzCopy manually:


     azcopy copy "https://source.blob.core.windows.net/container?SAS" "https://dest.blob.core.windows.net/container?SAS" --recursive=true

If the script fails, check AzCopy logs at:
```
     %USERPROFILE%\.azcopy\logs
```

References