question

NielsDeckers-7269 avatar image
0 Votes"
NielsDeckers-7269 asked ShaikMaheer-MSFT commented

How to run Azure Batch Pool Python script job serverless?

Hi all,

For a data pipeline between an API and my SQL-database.
I use an Azure Batch account, with a windows server 2019 (Virtual machine DSVM) and a pool with 2 dedicated nodes on a A1_v2 CPU.
I'm running this pipeline 3x per day to get data from the API through Python.
However my dedicated nodes are 90% of the day in idle state (not doing anything), do I still pay for these nodes the whole day?
If so, what would be a good solutions to cut costs on this batch account's nodes that i'm not using 90% of the time.

204991-image.png

thanks a lot in advance!


azure-data-factoryazure-batch
image.png (36.0 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @NielsDeckers-7269

Thanks for posting your query on Microsoft Q&A platform.

Yes, you will be billed even when the nodes are idle doing nothing. One option is to rescale the nodes to 0 when you are done with the task.

Please revert for questions.

Reference


1 Vote 1 ·
NielsDeckers-7269 avatar image NielsDeckers-7269 SubashriVasudevan-1752 ·

@SubashriVasudevan-1752

Thanks for the quick answer.
Do I need to rescale these nodes manually? or is this a setting somewhere?
Also i'm using 2 dedicated nodes, would 1 also be enough for this one Python script? (As I believe you pay per dedicated node)
I might have to use an Azure Function as this is serverless.

Thanks my man!

0 Votes 0 ·

Hey @NielsDeckers-7269 ,

No need to do manual rescale, you can use a web activity in a pipeline and follow the link I mentioned earlier to do the rescaling for you.

And as you mentioned, using Azure functions is a good alternative too but with one consideration, Azure functions time out in 10 minutes for consumption plan. (For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.)

Thank you.

0 Votes 0 ·
Show more comments

1 Answer

SubashriVasudevan-1752 avatar image
0 Votes"
SubashriVasudevan-1752 answered ShaikMaheer-MSFT commented

Hi @NielsDeckers-7269 ,

From the documentation

Multiple compute nodes: Individual nodes are not guaranteed to always be available. While uncommon, hardware failures, operating system updates, and a host of other issues can cause individual nodes to be offline. If your Batch workload requires deterministic, guaranteed progress, you should allocate pools with multiple nodes.

Hope that answers your question. Please feel free to accept it as answer ans/or up-vote for the benefit of community users.

Thank you.


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @NielsDeckers-7269 ,

Just checking in to see if the above answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·