Will AutoScale formula remove node that is running a task

Question

I am new to Azure Batch and I understand that it uses VMSS to create pools in the background. If I define an auto scale formula, can it result in a node being removed while a task is running?

Answer 1

Hello @AzureKnwGt-8579 

Thank you for reaching out to the Microsoft Q&A platform.

Looking at the title of this query, it appears you are inquiring about the Batch pool behavior after defining the auto-scale rule, a feature provided by the VMSS team (see here), rather than the auto-scale feature provided by the Batch team. Is this the scenario you are referring too?

If so, I assume the end-user has to configure the Batch account in user-subscription mode, allowing them to directly manage the VMSS resource (e.g., defining the auto-scale rule for VMSS) after it is provisioned by Batch. Therefore, the answer would differ from my previous response, which was based on the end-user using the auto-scale feature provided by the Batch team. 

Could you please provide the specific use case? This will help us answer your question more accurately.If I have answered your query, please click "Accept as answer" as a token of appreciation

Answer 2

Yes, it is possible for an autoscale formula to remove a node while a task is running. When automatic scaling decreases the number of compute nodes in a pool, Batch provides a node deallocation option that you can include in your formulas. You can specify that running tasks are stopped immediately and then requeued for execution on another node, or allowed to finish before the node is removed from the pool. However, setting the node deallocation option as taskcompletion or retaineddata will prevent pool resize operations until all tasks have completed, or all task retention periods have expired, respectively.

References:

• Nodes and pools in Azure Batch - Automatic scaling policy
• Create a formula to automatically scale compute nodes in a Batch pool - Autoscale formulas