How to run Azure Batch Pool Python script job serverless?

Niels Deckers 21

Hi all,

For a data pipeline between an API and my SQL-database.
I use an Azure Batch account, with a windows server 2019 (Virtual machine DSVM) and a pool with 2 dedicated nodes on a A1_v2 CPU.
I'm running this pipeline 3x per day to get data from the API through Python.
However my dedicated nodes are 90% of the day in idle state (not doing anything), do I still pay for these nodes the whole day?
If so, what would be a good solutions to cut costs on this batch account's nodes that i'm not using 90% of the time.

thanks a lot in advance!

Suba Balaji 11,186 Reputation points

2022-05-24T06:49:54.967+00:00

Hi @Niels Deckers

Thanks for posting your query on Microsoft Q&A platform.

Yes, you will be billed even when the nodes are idle doing nothing. One option is to rescale the nodes to 0 when you are done with the task.

Please revert for questions.

Reference
Niels Deckers 21 Reputation points

2022-05-24T07:10:51.723+00:00

@Suba Balaji

Thanks for the quick answer.
Do I need to rescale these nodes manually? or is this a setting somewhere?
Also i'm using 2 dedicated nodes, would 1 also be enough for this one Python script? (As I believe you pay per dedicated node)
I might have to use an Azure Function as this is serverless.

Thanks my man!
Suba Balaji 11,186 Reputation points

2022-05-24T07:26:14.03+00:00

Hey @Niels Deckers ,

No need to do manual rescale, you can use a web activity in a pipeline and follow the link I mentioned earlier to do the rescaling for you.

And as you mentioned, using Azure functions is a good alternative too but with one consideration, Azure functions time out in 10 minutes for consumption plan. (For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.)

Thank you.
Niels Deckers 21 Reputation points

2022-05-25T09:28:00.67+00:00

Hi @Suba Balaji

You're amazing, thanks so much for your help!
The last question I have is regarding dedicated nodes.
What's the difference in using 2 dedicated nodes (like in all tutorials) instead of one?
I don't really care if my job takes two times the time, but does it have other disadvantages?

Thanks again! ;)

1 answer

Suba Balaji 11,186 Reputation points

2022-05-25T10:15:30.387+00:00

Hi @Niels Deckers ,

From the documentation

Multiple compute nodes: Individual nodes are not guaranteed to always be available. While uncommon, hardware failures, operating system updates, and a host of other issues can cause individual nodes to be offline. If your Batch workload requires deterministic, guaranteed progress, you should allocate pools with multiple nodes.

Hope that answers your question. Please feel free to accept it as answer ans/or up-vote for the benefit of community users.

Thank you.
Please sign in to rate this answer.
ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2022-05-27T09:56:18.367+00:00

Hi @Niels Deckers ,

Just checking in to see if the above answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.
Sign in to comment