Azure Batch better handle Unusable nodes

Question

We have an Azure Batch pool with this Scale script:

startingNumberOfVMs = 10;
minVms = 0;
maxNumberofVMs = 1200;
maxAdditionalPerInterval = 200;

pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second);
pendingTaskSamples = pendingTaskSamplePercent < 2 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second));

desiredVms = min(maxNumberofVMs, pendingTaskSamples);
maxAllowed = $CurrentLowPriorityNodes + maxAdditionalPerInterval;
totalDesiredVms = min(desiredVms, maxAllowed);

$TargetDedicatedNodes = 0;
$TargetLowPriorityNodes = totalDesiredVms;
$NodeDeallocationOption = taskcompletion;

We queue batch jobs through the C# Batch library via CloudTasks, and we set MaxTaskRetryCount = 2 and MaxWallClockTime = TimeSpan.FromHours(12).

We have some tasks that failed on this pool where the task runs out of disk space (we're fixing the cause of that separately). The nodes that ran the failed jobs are marked as Unusable, while the jobs are marked as Active.

How do we set our scaling properly so failed nodes get deallocated and don't stay running -- is that $NodeDeallocationOption = requeue?

What do we need to do so that the jobs where nodes that run out of space show with an error instead of being marked as Active? It seems like maybe they are set to be retried, but there aren't any usable nodes, because the number of Unusable nodes is equal to the $TargetLowPriorityNodes count. Do we need to add the number of Unusable nodes to our $TargetedLowPriorityNodes count?

Answer 1

• For the $NodeDeallocationOption question, my understanding is Microsoft is working on improving the documentation to answer my question.- For the Unusable nodes question, there is a bug that Microsoft is fixing in the $UsableNodeCount variable. It currently includes Unusable nodes because while the disk is Unusable the node is still currently considered "usable" since it responds to queries. Once Microsoft has fixed this issue, the following script will account for the Unusable nodes:

    $TargetDedicatedNodes = 0;
    $NodeDeallocationOption = taskcompletion; // Keep the nodes active until the tasks finish
    maxNodes = 1200;
    // https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling#variables
    pendingTaskSamples = max($PendingTasks.GetSample(TimeInterval_Minute * 3)); // Max number of pending tasks (i.e. active tasks (ready to execute but not yet executing) and running tasks) over the last 3 minutes
    usableNodeCount = max($UsableNodeCount.GetSample(TimeInterval_Minute * 1)); // Cannot use $UsableNodeCount.GetSample(1) because the variable has a time delay and the most recent entry is unreliable
    unusableNodeCount = $CurrentDedicatedNodes + $CurrentLowPriorityNodes - usableNodeCount;
    $TargetLowPriorityNodes = min(maxNodes, pendingTaskSamples + unusableNodeCount);
  

Answer 2

• For the $NodeDeallocationOption question, my understanding is Microsoft is working on improving the documentation to answer my question.- For the Unusable nodes question, there is a bug that Microsoft is fixing in the $UsableNodeCount variable. It currently includes Unusable nodes because while the disk is Unusable the node is still currently considered "usable" since it responds to queries. Once Microsoft has fixed this issue, the following script will account for the Unusable nodes:

    $TargetDedicatedNodes = 0;
    $NodeDeallocationOption = taskcompletion; // Keep the nodes active until the tasks finish
    maxNodes = 1200;
    // https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling#variables
    pendingTaskSamples = max($PendingTasks.GetSample(TimeInterval_Minute * 3)); // Max number of pending tasks (i.e. active tasks (ready to execute but not yet executing) and running tasks) over the last 3 minutes
    usableNodeCount = max($UsableNodeCount.GetSample(TimeInterval_Minute * 1)); // Cannot use $UsableNodeCount.GetSample(1) because the variable has a time delay and the most recent entry is unreliable
    unusableNodeCount = $CurrentDedicatedNodes + $CurrentLowPriorityNodes - usableNodeCount;
    $TargetLowPriorityNodes = min(maxNodes, pendingTaskSamples + unusableNodeCount);