Sync To Blob using a PS Script

Question

Hi,

When I manually run the PS script to sync from source to destination (Blob Container) it identifies the destination endpoint perfectly fine. The script is able to identify changes on the source and compare against the destination (Blob). However, when the script is run using Task Scheduler, I receive the below message. I have a trailing / at the end. I also have the latest and greatest azcopy.exe.

---

Transcript started, output file is C:_DEV\BlobSync.txt

Cannot perform sync due to error: trying to sync between different resource types (either file <->directory or directory <-> file) which is not allowed.sync must happen between source and destination of the same type, e.g. either file <-> file or directory <-> directory.To make sure target is handled as a directory, add a trailing '/' to the target.

Here is the script.

Start-Transcript -Path "C:_DEV\BlobSync.txt" -Append

$env:AZCOPY_CONCURRENCY_VALUE = "AUTO"

C:_DEV\azcopy.exe sync "H:\test1" "https://***.blob.core.windows.net/hannover/zkwnas2/test1/?sv=" --recursive --log-level=INFO

$env:AZCOPY_CONCURRENCY_VALUE = ""

Stop-Transcript

Answer 1

I resolved this. It was my own fault. The token expired. Having said that I did notice additional issues and wanted to mention what I did to resolve them. It also appears from a performance perspective you want to use azcopy.exe copy command instead of sync. We have tens of thousand of small files. I also defined the parameter --overwrite ifsourcenewer

"The sync command identifies all files at the destination, and then compares file names and last modified timestamps before the starting the sync operation. If you have a large number of files, then you can improve performance by eliminating this up-front processing.

To accomplish this, use the azcopy copy command instead, and set the --overwrite flag to ifSourceNewer. AzCopy will compare files as they are copied without performing any up-front scans and comparisons. This provides a performance edge in cases where there are a large number of files to compare."

When using the Task Scheduler the network mapping (H:\test1) in the PS script would not run. It did work when manually running the script. To allow Task Scheduler to run the script correctly I needed to use a UNC path instead of the mapping. Because I changed to a copy command the path to Blob needed the test1 container removed. Not doing so created another test1 container in the test1 container. https://*.blob.core.windows.net/hannover/zkwnas2/

Answer 2

@Taylor, Kevin J

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: When I run the PS script to sync from source to destination (Blob Container) using Task Scheduler, I receive the error message: Cannot perform sync due to error: trying to sync between different resource types (either file <->directory or directory <-> file) which is not allowed.sync must happen between source and destination of the same type, e.g. either file <-> file or directory <-> directory.To make sure target is handled as a directory, add a trailing '/' to the target.

Cause & Solution: Cx shared - I resolved this. It was my own fault. The token expired. Having said that, I did notice additional issues and wanted to mention what I did to resolve them. It also appears from a performance perspective you want to use azcopy.exe copy command instead of sync. We have tens of thousands of small files. I also defined the parameter --overwrite ifsourcenewer

The sync command identifies all files at the destination, and then compares file names and last modified timestamps before starting the sync operation. If you have many files, then you can improve performance by eliminating this up-front processing.

To accomplish this, use the azcopy copy command instead, and set the -overwrite flag to ifSourceNewer. AzCopy will compare files as they are copied without performing any up-front scans and comparisons. This provides a performance edge in cases where there are many files to compare.

When using the Task Scheduler, the network mapping (H:\test1) in the PS script would not run. It worked manually running the script. To allow Task Scheduler to run the script correctly I needed to use a UNC path instead of the mapping. Because I changed to a copy command the path to Blob needed the test1 container removed. Not doing so created another test1 container in the test1 container. https://*.blob.core.windows.net/hannover/zkwnas2/

---

If your issue remains unresolved or have further questions, please let us know in the comments how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback.