Azure Batch - Nodes will not leave the pool (stuck in leaving pool state)

Brian Bertrand 21

We have had a batch system running for years. Recently we have seen issues where our node pools will scale up no problem, but will fail to scale down for hours stuck in 'leaving' state.

This is currently happening and I have no way to remove the nodes without deleting the pool (I suspect even this will fail).

Has anyone ever encountered this? I'm worried our company is being charged for machines we aren't using which is extremely alarming when using a fully automated system.

Edit: The nodes finally left the pool after 4 hours. I'm still interested as to why this occurs though, and if the time the node is in leavingpool state costs money to the customer.

Nikhil Duserla 2,825 Reputation points Microsoft Vendor

2024-06-19T08:30:13.9233333+00:00

Hi Brian Bertrand ,

Thanks for reaching out to us on the Microsoft Q&A forum.

We understand from your query that you are experiencing an issue where your Azure Batch node pools are scaling up but not scaling down.

It is likely due to an error in the auto-scaling formula. The error indicates that there's an evaluation error in the formula, specifically with the timestamp.

Azure Batch autoscaling only supports precision up to seconds, but the formula you might using a more precise timestamp.

Please modify the auto-scaling formula to remove the timestamp and use a different approach and work with this formula.

$samples = $PendingTasks.GetSamplePercent(TimeInterval_Minute * 15);

$tasks = $samples < 70 ? max(0,$PendingTasks.GetSample(1)) : max( $PendingTasks.GetSample(1), avg($PendingTasks.GetSample(TimeInterval_Minute * 15)));

$targetVMs = $tasks > 0? $tasks:max(0, $TargetDedicatedNodes/2);

$TargetDedicatedNodes = max(0, min($targetVMs, 20));

$NodeDeallocationOption = taskcompletion;

This formula uses a simpler approach to calculate the target node count based on pending tasks, without relying on a precise timestamp.

If you have any further queries, do let us know.

If the answer is helpful, please click "Accept Answer" and "Upvote it"

Share via

Azure Batch - Nodes will not leave the pool (stuck in leaving pool state)

Your answer