Python in ADF: How to choose the best approach?

Question

I have a simple python script that does some data processing that I want to run from within an ADF pipeline. I can see that there are multiple ways to do this:

• Azure Function
• Azure Batch Pool
• Databricks
• Python runbook

I am starting from scratch as we do not currently have any batch pools or Azure functions set up, and are not using Databricks. I am struggling to determine which is the best approach for my usecase. Is there somewhere I can find a list of pros and cons for each method to help decide?

(Current usecase is simple data manipulation using Pandas; will be reading/writing to Azure SQL databases; doesn't require a large compute resource; will only be run intermittently; needs to be called from an ADF pipeline).

Answer 1

@ellishar

Thanks for reaching out to Q&A.

All four services can be called from Azure ADF pipeline and they have the ability to run python script. I would suggest that you choose between Azure functions and python runbook as they are cost effective when compared to Databricks and Batch. Azure functions serverless tier can scale automatically and doesnt cost much for the load you will experience, moreover SQL integration would be simple with Functions and python runbook.

So you can choose between Azure Functions and runbook.