Automate file transfer from REST API to Azure Blob Storage

Question

I need to automate the download of several files (up to 100MB each) from a REST API endpoint, and then upload them to Azure Blob Storage. I need to do this using only Azure services (no local/on premises resources) on a scheduled basis.

What Azure service would you recommend to accomplish this task? I would prefer to use a serverless/usage-based billed service. I have looked at Logic Apps, Functions, Data Factory and Automation as possible options, but I need guidance on which service would be a better fit considering the files' size (up to 100MB) while keeping cost as low as possible.

Answer 1

Each of the services you've mentioned are all good solutions to implement. And all these services have consumption-based pricing, though they do cost different based on the service.

For Logic Apps, there is a message size limit of 100MB. So, if there are any files above 100MB (at some point), you will have to support chunking on your REST API.

For Azure Functions, you could implement a durable function that could be a long running job to copy all files one by one. Depending on how fast files are downloaded from your Rest API, you will have to make sure to choose the right plan.

For Azure Data Factory, there is an HTTP Connector and Blob Storage Connector that you could directly use. Of the lot, this is service is purpose built for scenarios such as yours.

For Azure Automation, you could write a runbook using PowerShell to achieve the same using the relevant cmdlets.

In terms of implementation, I believe ADF would be simpler.

In terms of cost, Azure Functions with its free tier might be a good fit for smaller volumes. But with higher volumes, you could get an estimate using Azure Pricing Calculator.