AzCopy Checksum Verification when both Source and Destination are Remote

Balaram 1 Reputation point
2022-02-11T15:46:04.12+00:00

Hi, I am transferring files from S3 to azure and I would like to verify the integrity of files when I download these on my local. For this, I have set the --put-md5 flag to true while copying but azcopy has returned error saying checksum creation fails for S2S transfers. Is there a way one could verify the consistency of files when both source and destination are remote? I have attached the error returned as screenshot.
Regards,
Balaram.173614-inkedinkedscreenshot-2022-01-17-144037-li.jpg

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,415 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. deherman-MSFT 33,141 Reputation points Microsoft Employee
    2022-02-11T22:50:22.547+00:00

    @Balaram
    I understand you are wanting to download your S3 files locally verify the checksum then upload them to blob storage using AzCopy also verifying the checksum after upload. When utilizing AzCopy between S3 and Azure Storage the files are copied directly and are not downloaded locally, which is why the --put-md5 flag is not an option. You would need to download the files locally, verify the checksum, then upload using AzCopy --put-md5 if you wish to add a Content-md5 property to your blob.

    However, given AzCopy has made its best effort to protect data integrity during transferring, the validation step above is probably redundant and not recommended unless data integrity is much more important than performance. You should be able to safely transfer data directly from S3 to Azure Storage without issues.

    AzCopy creates log and plan files for every job. You can use the logs to investigate and troubleshoot any potential problems.
    The logs will contain the status of failure (UPLOADFAILED, COPYFAILED, and DOWNLOADFAILED), the full path, and the reason of the failure.
    By default, the log and plan files are located in the %USERPROFILE.azcopy directory on Windows or $HOME.azcopy directory on Mac and Linux.

    -------------------------------

    Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments