How do I get an Azure Batch pool to resubmit after more cores become available?

Adam Talbot 1

I'm using AutoPools with Jobs in Azure Batch, therefore I have a 1-to-1 ratio of Pools to Jobs.

If I submit a few jobs at the same time, one of them may be prevented from starting by the core quota on the Azure Batch account, i.e. the AutoPool would cause the number of cores to exceed the limit. When this happens the Job will never start and instead sit idle until I manually resize the pool.

Is there a way for the Job and AutoPool to resubmit once the number of cores in use has reduced and therefore there is free space? Could I use an Autoscaling Formula with a fixed number to resize the pool every x minutes?

prmanhas-MSFT 17,906 Reputation points Microsoft Employee

2020-08-25T11:15:34.97+00:00

@Adam Talbot Apologies for the delay in response and all the inconvienince caused because of the issue.
I have reached out to our internal team on the issue and once I have an update will keep you posted on same.

Thank you for your patience over the matter.

Thanks.
Adam Talbot 1 Reputation point

2020-08-26T16:51:57.257+00:00
Thank you.

I used an autoscale formula with a fixed amount, e.g:

$NodeDeallocationOption = taskcompletion; $TargetDedicatedNodes = 2; $TargetLowPriorityNodes = 0;

Which seemed to enable pools to fill the quota where available, then queue after that. However this only worked briefly before another problem appeared, now all of my pools get stuck in resizing, which is likely to be a separate issue.

I would be grateful if you could confirm whether using autoscaling would enable pool queuing while identify what is causing the other issue.

1 answer

prmanhas-MSFT 17,906 Reputation points Microsoft Employee

2020-08-27T06:54:00.703+00:00

@Adam Talbot Thank you for your response. I had discussion internally and they did mentioned that using an Autoscaling Formula with a fixed number to resize the pool every x minutes will be the solution to olve your issue.

Do let me know in case of any queries.

Thanks
Please sign in to rate this answer.
prmanhas-MSFT 17,906 Reputation points Microsoft Employee

2020-09-01T15:22:59.803+00:00

@Adam Talbot Any update on the issue?

If the suggested response helped you resolve your issue, do click on "Mark as Answer" and "Up-Vote" for the answer that helped you for benefit of the community.

Thanks
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How do I get an Azure Batch pool to resubmit after more cores become available?

1 answer

Your answer