Copying data to Azure Blob from Amazon S3 bucket.

Vinay5 41 Reputation points


I have a requirement to copy data from Amazon S3 to Azure BloB. There are 10 objects in this bucket and each objects has files in the below format.
Month Folder> Date Folder> Day Folder< Hour Folder . For example the files for 2021-03-23 (at 17:00(time)) are present in the folder path 2021<03<23<17.

The files are placed in S3 bucket daily. I have to copy only the latest files placed in S3 to Azure. So, as in when the files are placed at particular time window in S3 ,only the latest delta files from that particular object should be copied. In this way, I should be able to copy files from all the objects. Could you please let me know how to achieve this.

Thank you.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,210 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,292 questions
{count} votes