Azure Function

Question

I would like to hear your thoughts whether Azure Function is the best solution to our application. If not, which solution we should use on Azure.

Our application is a data pipeline fetching data periodically (~15min) from multiple sources (API, client database) and reformat them and then ingest into our Azure SQL database. Right now, the entire data pipeline is just several Python script running in a Windows VM on Azure.

Two sub-pipelines I was trying to migrate to Azure Functions using two time triggers look like below. AgriNet, AzureSQLCommunications and Farmbeats are user-defined libraries that handle communications with provider API/database and Azure SQL database. Is Azure Fucntions is the solution to go!

Answer 1

Azure Functions is indeed a good option, especially considering that you already have python scripts. It should be fairly straightforward to move them into functions.

But depending on how long each invocation is expected to run, you would have to choose your plan accordingly for the timeout limit that the consumption tier has.

Another option would be to Azure Data Factory which provides a visual pipeline editor to create your data pipelines. You could find all available connectors in the docs.