Automate file transfer from REST API to Azure Blob Storage

Mario VW 1 Reputation point
2020-10-17T18:57:45.933+00:00

I need to automate the download of several files (up to 100MB each) from a REST API endpoint, and then upload them to Azure Blob Storage. I need to do this using only Azure services (no local/on premises resources) on a scheduled basis.

What Azure service would you recommend to accomplish this task? I would prefer to use a serverless/usage-based billed service. I have looked at Logic Apps, Functions, Data Factory and Automation as possible options, but I need guidance on which service would be a better fit considering the files' size (up to 100MB) while keeping cost as low as possible.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,262 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,843 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions
Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,114 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Pramod Valavala 20,516 Reputation points Microsoft Employee
    2020-10-20T11:29:32.627+00:00

    Each of the services you've mentioned are all good solutions to implement. And all these services have consumption-based pricing, though they do cost different based on the service.

    For Logic Apps, there is a message size limit of 100MB. So, if there are any files above 100MB (at some point), you will have to support chunking on your REST API.

    For Azure Functions, you could implement a durable function that could be a long running job to copy all files one by one. Depending on how fast files are downloaded from your Rest API, you will have to make sure to choose the right plan.

    For Azure Data Factory, there is an HTTP Connector and Blob Storage Connector that you could directly use. Of the lot, this is service is purpose built for scenarios such as yours.

    For Azure Automation, you could write a runbook using PowerShell to achieve the same using the relevant cmdlets.

    In terms of implementation, I believe ADF would be simpler.

    In terms of cost, Azure Functions with its free tier might be a good fit for smaller volumes. But with higher volumes, you could get an estimate using Azure Pricing Calculator.

    0 comments