Azure batch autoscale formula

Question

Azure batch autoscale formula

Gal Hadar 0

I'm trying to create a pool that always starts with 1 node. After 3 minutes of having more than 70% CPU usage the pool should generate another machine, and repeat. I'm using the current formula:

initialPoolSize = 1;
$TargetDedicatedNodes = initialPoolSize;
$totalDedicatedNodes =
    (avg($CPUPercent.GetSample(TimeInterval_Minute * 3)) >= 0.7) ?
    ($TargetDedicatedNodes + 1) : $TargetDedicatedNodes;

I also tried using this formula that is posted in the Microsoft documentation:

initialPoolSize = 1;
$TargetDedicatedNodes = initialPoolSize;
$totalDedicatedNodes =
    (min($CPUPercent.GetSample(TimeInterval_Minute * 3)) >= 0.7) ?
    ($CurrentDedicatedNodes * 1.1) : $CurrentDedicatedNodes;

Both formulas don't work, I'm stuck with only 1 node for the entire build time, and the CPU usage is at 100%.

If anyone has any experience with this and or can help that'll be greatly appreciated.

vipullag-MSFT 19,066 Reputation points Microsoft Employee

2023-05-25T04:39:36.9833333+00:00
Hello Gal Hadar

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Based on the formulas you provided, it seems that you are trying to create an autoscale formula that starts with one node and adds another node if the CPU usage is above 70% for three minutes. However, there are some issues with the formulas you provided.

In the first formula, you are setting the $totalDedicatedNodes variable to either $TargetDedicatedNodes + 1 or $TargetDedicatedNodes, depending on whether the CPU usage is above 70%. However, you are not actually setting the $TargetDedicatedNodes variable to the new value of $totalDedicatedNodes. Therefore, the pool size will never change.

initialPoolSize = 1; $TargetDedicatedNodes = initialPoolSize; $totalDedicatedNodes = (avg($CPUPercent.GetSample(TimeInterval_Minute * 3)) >= 0.7) ? ($TargetDedicatedNodes + 1) : $TargetDedicatedNodes; $TargetDedicatedNodes = $totalDedicatedNodes;

This formula sets the $totalDedicatedNodes variable to either $TargetDedicatedNodes + 1 or $TargetDedicatedNodes, depending on whether the CPU usage is above 70%. Then, it sets the $TargetDedicatedNodes variable to the new value of $totalDedicatedNodes, which will cause the pool size to increase if necessary.

Here is a modified version of the second formula that should work for your scenario:

initialPoolSize = 1; $TargetDedicatedNodes = initialPoolSize; $CurrentDedicatedNodes = $TargetDedicatedNodes + $TargetLowPriorityNodes; $TotalCpu = avg($CPUPercent.GetSample(TimeInterval_Minute * 3)); $Threshold = 0.7; $ScaleUpFactor = 1.1; if ($TotalCpu >= $Threshold) { $NewNodeCount = $CurrentDedicatedNodes * $ScaleUpFactor; $NewNodeCount = min($NewNodeCount, $MaxNodes); $NewNodeCount = int($NewNodeCount); $TargetDedicatedNodes = $NewNodeCount; }

This formula calculates the average CPU usage over the last 3 minutes and compares it to a threshold value of 0.7. If the average CPU usage is greater than or equal to the threshold, it calculates a new node count based on the current number of nodes and a scale-up factor of 1.1. It then sets the target number of dedicated nodes to the new node count, subject to a maximum number of nodes. The formula assumes that you have defined the $MaxNodes variable somewhere in your code.

I hope this helps!
Gal Hadar 0 Reputation points

2023-05-28T08:59:55.91+00:00

It's interesting that you say that there's something wrong with the formulas I posted because I took them straight from the official Microsoft Azure batch documentation.

I wonder if I'm looking at this wrong when I'm talking about CPU utilization, I'm guessing Azure Batch is looking at at the utilization from task manager --> performance tab

Edit: it seems the first formula works but it only scales up 1 more node and is stuck at 2 nodes for the rest of the build. I don't know if it's something with the formula or my batch account...
vipullag-MSFT 19,066 Reputation points Microsoft Employee

2023-05-29T04:40:52.56+00:00
Hello Gal Hadar

Thanks for responding back and confirming that you were able to scale a node up using the shared formula.

However, we noticed your have rated low on one of the answer as not helpful. Thank you for taking time to share feedback.

We are here to help you and strive to make your experience better and greatly value your feedback.

Based on your recent comment and information provided, if the autoscale formula is only scaling up to 2 nodes and then stopping, it's possible that there is a limit on the number of nodes that can be created in your Batch account. Requesting you to check the quotas and limits for your Batch account

If you have reached the limit for the number of nodes that can be created in your Batch account, you will need to request a quota increase.

If you have not reached the limit for the number of nodes that can be created in your Batch account, try modifying the formula to scale up more aggressively by increasing the number of nodes added when the CPU usage is above 70%. For example, you could try the following formula which adds 2 nodes when the CPU usage is above 70%, which should cause the pool size to increase more quickly.

initialPoolSize = 1; $TargetDedicatedNodes = initialPoolSize; $totalDedicatedNodes = (avg($CPUPercent.GetSample(TimeInterval_Minute * 3)) >= 0.7) ? ($TargetDedicatedNodes + 2) : $TargetDedicatedNodes; $TargetDedicatedNodes = $totalDedicatedNodes;
Gal Hadar 0 Reputation points

2023-05-29T12:58:26.0966667+00:00

So regarding my quota, I have not reached my limit.

Regarding the second part, that's not what I mean. I want the formula to keep adding machines if the CPU utilization is above 70%, but because $TargetDedicatedNodes = initialPoolSize and initialPoolSize = 1, technically $TargetDedicatedNodes = 1. If the formula proves true it'll just keep adding 1 to 1 which is why the formula is stuck at 2 nodes and won't generate more

Gal Hadar 0 Reputation points

2023-05-28T08:59:55.91+00:00

It's interesting that you say that there's something wrong with the formulas I posted because I took them straight from the official Microsoft Azure batch documentation.

I wonder if I'm looking at this wrong when I'm talking about CPU utilization, I'm guessing Azure Batch is looking at at the utilization from task manager --> performance tab

Edit: it seems the first formula works but it only scales up 1 more node and is stuck at 2 nodes for the rest of the build. I don't know if it's something with the formula or my batch account...
vipullag-MSFT 19,066 Reputation points Microsoft Employee

2023-05-29T04:40:52.56+00:00

Hello Gal Hadar

Thanks for responding back and confirming that you were able to scale a node up using the shared formula.

However, we noticed your have rated low on one of the answer as not helpful. Thank you for taking time to share feedback.

We are here to help you and strive to make your experience better and greatly value your feedback.

Based on your recent comment and information provided, if the autoscale formula is only scaling up to 2 nodes and then stopping, it's possible that there is a limit on the number of nodes that can be created in your Batch account. Requesting you to check the quotas and limits for your Batch account

If you have reached the limit for the number of nodes that can be created in your Batch account, you will need to request a quota increase.

If you have not reached the limit for the number of nodes that can be created in your Batch account, try modifying the formula to scale up more aggressively by increasing the number of nodes added when the CPU usage is above 70%. For example, you could try the following formula which adds 2 nodes when the CPU usage is above 70%, which should cause the pool size to increase more quickly.

initialPoolSize = 1; $TargetDedicatedNodes = initialPoolSize; $totalDedicatedNodes = (avg($CPUPercent.GetSample(TimeInterval_Minute * 3)) >= 0.7) ? ($TargetDedicatedNodes + 2) : $TargetDedicatedNodes; $TargetDedicatedNodes = $totalDedicatedNodes;
Gal Hadar 0 Reputation points

2023-05-29T12:58:26.0966667+00:00

So regarding my quota, I have not reached my limit.

Regarding the second part, that's not what I mean. I want the formula to keep adding machines if the CPU utilization is above 70%, but because $TargetDedicatedNodes = initialPoolSize and initialPoolSize = 1, technically $TargetDedicatedNodes = 1. If the formula proves true it'll just keep adding 1 to 1 which is why the formula is stuck at 2 nodes and won't generate more

Answer 1

Sedat SALMAN 5,245

startingNumberOfVMs = 1;
maxNumberOfVMs = 20; // maximum number of VMs you want
samplePercentThreshold = 0.7;
sampleDuration = TimeInterval_Minute * 3; 
// Get the last sample
$sample = $CPUPercent.GetSamplePercent(sampleDuration);
// If the average CPU usage was more than 70%, increment the number of nodes
$TargetDedicated = ($sample > samplePercentThreshold ? startingNumberOfVMs + 1 : startingNumberOfVMs);
// Always keep the number of nodes between the starting number and the maximum
$TargetDedicated = max(startingNumberOfVMs, min($TargetDedicated, maxNumberOfVMs));

Gal Hadar 0 Reputation points

2023-05-28T09:01:25.69+00:00

Someone from StackOverflow recommended me this exact same formula and unfortunately it does not work. The $sample variable always puts out 100 even though the actual CPU percentage is at around 6% in the machines task manager --> performance, I'm guessing that's where Azure batch looks to decide CPU utilization %

Answer 2

This is the answer for anyone wondering

maxNumberOfVMs = 20; // maximum number of VMs you want

samplePercentThreshold = 0.7;

sampleDuration = TimeInterval_Minute * 3;

// Get the last sample

$sample = (avg($CPUPercent.GetSample(sampleDuration)));

// If the average CPU usage was more than 70%, increase nodes by one, if not true keeps as is

$TargetDedicated = ($sample >= samplePercentThreshold ? $TargetDedicated +1 : $TargetDedicated);

// If the average CPU usage is below 20% decrease nodes by one, if not true keep as is

$TargetDedicated = ($sample <= 0.2 ? $TargetDedicated - 1 : $TargetDedicated);

// Always keep the number of nodes under the maximum

$TargetDedicated = min($TargetDedicated, maxNumberOfVMs);

Azure batch autoscale formula

2 answers

Activity