AZCopy Sync Skipping Zero Byte Files in Azure Blob Storage

Question

I am attempting to replicate Azure blob storage to another subscription's blob storage using AZCopy Sync. The process works well, but it skips certain files.

To represent empty folders, dummy files (0 bytes) are created with the name <no name>. However, with AZCopy Sync, all the 0-byte <no name> files are not being copied, resulting in the absence of empty folders in the destination.

Any suggestions would be appreciated.

Answer 1

Hi @Vaibhav Jain

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: AZCopy Sync Skipping Zero Byte Files in Azure Blob Storage.

Solution: Issue has been resolved by using include-directory-stub flag.

azcopy sync "source" "destination" --recursive --delete-destination true --include-directory-stub=true.

If your issue remains unresolved or have further questions, please let us know in the comments how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback.

Please let us know if you have any further queries. I’m happy to assist you further. 

---

Please do consider to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello Vaibhav Jain,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your AZCopy Sync Skipping Zero Byte Files in Azure Blob Storage.

Others have reported similar issues with 0-byte files being skipped - https://github.com/Azure/azure-storage-azcopy/issues/1291. They resolved it by using workarounds like the ones mentioned below. If you can implement these strategies, you can effectively replicate Azure Blob Storage while preserving empty folders using dummy files. Whether you adjust AZCopy configurations or incorporate complementary tools and scripts, these solutions provide practical steps to overcome the limitations of AZCopy Sync.

1. To ensure that dummy files are included in the sync operation, you can use the AZCopy --include-pattern flag. This flag allows you to specify patterns for files that should be included in the sync process. For example using bash:

azcopy sync <source-path> <destination-path> --include-pattern "<no name>"

This command ensures that even 0-byte dummy files with the name <no name> are included in the sync operation.

2. If the --include-pattern flag does not work as expected and <no name> files are still being skipped, you can use a script to preprocess these files. Convert 0-byte files to 1-byte files before syncing in bash command: find <source-path> -type f -empty -exec truncate -s 1 {} \;

After syncing, you can revert these files back to 0 bytes at the destination if necessary:

find <destination-path> -type f -exec truncate -s 0 {} \;

This approach ensures that all files, including those representing empty folders, are copied.

3. Consider using Azure Data Factory or a similar ETL tool for more robust storage replication as alternative tool. These tools offer advanced options for preserving folder structures and handling 0-byte files. For instance, you can set up a Copy Data pipeline in Azure Data Factory to replicate blobs while maintaining metadata, such as folder representations.
4. Another approach is to generate a metadata file before syncing. This file can explicitly instruct AZCopy to include and create folders. Use the following command to generate the metadata file using bash: azcopy list <source-path> --recursive=true --output <metadata-file>

Then, apply this metadata during the sync to ensure all folders, represented by dummy files, are included.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or need alternative solution.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Answer 3

I have figured out the solution by using include-directory-stub flag.

azcopy sync "source" "destination" --recursive --delete-destination true --include-directory-stub=true

Thank you so much for your suggestions.