Share via

How to fetch files from S3 to azure blob using logic app

Vivek Komarla Bhaskar 956 Reputation points
2023-04-05T15:50:19.0966667+00:00

On my S3 bucket, I have a folder structure similar to this -  Year -> Month -> Day -> Hour I receive files in .gz format every minute on S3. How can I push these files to ADLS gen2 through logic apps? Also, when the file is pulled from the S3 bucket, it must be unzipped Screenshot 2023-04-06 at 9.07.21 am

Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
3,548 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,632 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Pramod Valavala 20,656 Reputation points Microsoft Employee Moderator
    2023-04-19T16:00:36.3166667+00:00

    Based on the scenario that you've described, you are better off using Azure Data Factory (ADF) instead, which is designed to move and transform data.

    With ADF, you will have to use the following connectors, and like Logic Apps, build a visual pipeline.

    1. Binary Dataset with compression settings as required (handled decompression)
    2. Amazon S3 Connector
    3. Copy Activity
    4. ADLS Gen2 Connector

    You can refer to this solution doc that covers migrating data from Amazon S3 to Azure Storage as a starting point.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.