Sync data between AWS S3 and azure Blob

Jayesh Sawant 76 Reputation points
2022-10-07T13:57:25.92+00:00

Hi Team,

I am looking for solution where data in AWS S3 and Azure Blob should be in sync so that if new files added or updated in source then it should reflected or sync in destination and vice versa both.

Please let us know whether AZ-copy will sync in this both way scenario ?
is there any tool or service available to sync data in both source and destination ?

Please help...

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,775 questions
{count} vote

3 answers

Sort by: Most helpful
  1. JohnyBenz 306 Reputation points
    2022-10-21T12:03:10.697+00:00

    Azcopy should does this job if you have the right code but it will not work automatically or in scheduled time. And as a straightforward answer to your question, there are GUI tools that have the ability to copy /sync to/from Azure blob/AWS S3 , or to keep both in sync, for example, Gs Richcopy360 and Goodsync, also both can copy the updates files only, both can transfer between different clouds storage like from Azure blob to S3, from S3 to Azureblob, from Azure blob to Wasabi.........etc.
    Hope my answer helps

    1 person found this answer helpful.
    0 comments No comments

  2. PRADEEPCHEEKATLA-MSFT 80,251 Reputation points Microsoft Employee
    2022-10-10T11:53:39.27+00:00

    Hello @Jayesh Sawant ,

    Thanks for the question and using MS Q&A platform.

    Unfortuantely, there is no specific tool or the service available to sync data in both AWS S3 and Azure Blob in Azure.

    As per the research I had found third-party article which explains about the Multi-cloud connection: Transfer files between Amazon S3, Azure Blob and Google Cloud Storage using SFTP Gateway

    Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
    0 comments No comments

  3. Takahito Iwasa 4,841 Reputation points MVP
    2022-10-11T01:25:58.703+00:00

    Hi, @Jayesh Sawant

    As @PRADEEPCHEEKATLA-MSFT said, there is no bi-directional synchronization tool.

    I will guide you through the results of my past testing and possible approaches that can be implemented.
    The following article is a verified article when I tried to do the same thing before. (Sorry it's in Japanese. Please translate)

    https://dev.classmethod.jp/articles/azure-storage-azcopy-amazon-s3-datasource/

    First of all, AzCopy allows him to specify S3 as the source, but not as the target.
    This was also discussed in an Issue, but it seems there are no plans to implement it.

    https://github.com/Azure/azure-storage-azcopy/issues/348

    However, on AWS he shows how to use azure-blob-to-s3 for Blob as source and S3 as target.

    https://aws.amazon.com/jp/blogs/storage/one-way-to-migrate-data-from-azure-blob-storage-to-amazon-s3/

    Therefore, one way for you to achieve this without relying on third-party tools is to implement separate one-way synchronizations from AWS to Azure and from Azure to AWS. (You may also need to think about how to consider conflicts, etc.)

    0 comments No comments