Machine Learning: Unable to Launch Multi-Node GPU Compute Cluster Despite Having Adequate Quota

Question

Problem description

I am trying to launch a multi-node GPU compute cluster in Azure Machine Learning with the H100 v5 VM size in the 'southafricanorth' region. Although the portal shows sufficient quota for the requested resources, the cluster fails to provision, giving an internal execution error and stating that the desired number of dedicated nodes could not be allocated. I have tried creating a 1-node cluster by setting min=0 and max=1, which succeeded but resulted in zero nodes. When I set min=1 and max=1, it failed with an internal execution error. I also confirmed that I can create single-node H100 v5 instances separately, but the multi-node cluster fails. I have not received specific provisioning error messages or codes, and I am unsure if this is due to regional capacity constraints, quota misconfigurations, or a provisioning issue. I am seeking guidance on how to resolve this provisioning failure.

Environment

Azure Machine Learning managed compute cluster in 'southafricanorth' region, requesting 2 H100 v5 GPU nodes, approximately 192 cores.

What I've already tried

I checked the quota for H100 v5 resources in the portal and confirmed that it appeared sufficient for my request. I created single-node H100 v5 compute instances successfully, but when attempting to create a multi-node cluster with min=1 and max=1, it failed with an internal execution error. I also tried deleting previous clusters and instances before testing. No detailed error messages or diagnostics were available beyond the internal execution error message.

Current status

The cluster fails to provision with an internal execution error, even though quota seems sufficient. I am looking for steps to diagnose whether this is a regional capacity issue, quota misconfiguration, or a provisioning bug, and how to resolve the failure to create the multi-node GPU cluster.

Answer 1

Hi ng-1043 & thx for join me at Q&A platform,

this looks more like an Azure ML compute allocation issue than a straightforward quota problem, although the underlying provisioning error is needed to confirm the cause.

The important distinction is that having enough GPU quota doesn't guarantee that Azure has the physical capacity available to allocate those VMs. H100 v5 instances are particularly capacity-sensitive, and multi-node deployments can have additional placement requirements, especially when high-speed InfiniBand connectivity is involved.

One thing worth clarifying: when u create a cluster with min=0 and max=1, Azure ML can successfully create the compute resource without allocating any VM nodes. So seeing zero nodes in that configuration is expected and doesn't prove that H100 capacity is available.

What stands out is that even min=1, max=1 fails, while a standalone H100 compute instance works. That suggests the issue isn't necessarily limited to multi-node placement. Azure ML compute instances and managed compute clusters use different provisioning workflows, so success with one doesn't guarantee the other can allocate the same SKU.

I'd first verify the exact VM SKU, the available dedicated VM-family quota, and the total regional vCPU quota in South Africa North. Make sure the requested cores aren't already consumed by other resources and that the quota applies to dedicated nodes rather than low-priority compute.

Next, check the failed cluster's provisioning state and error details in Azure ML Studio, along with the Azure Activity Log for the corresponding resource operations. If the error is still just InternalExecutionError or something like unable to allocate the desired number of dedicated nodes, that's not enough to distinguish a capacity shortage from an internal Azure ML provisioning failure.

I'd also try creating a single-node compute cluster with the same SKU in another supported region, assuming that's acceptable for ur workload. If it works elsewhere but consistently fails in South Africa North, that would strengthen the regional capacity hypothesis. For a two-node H100 cluster, Azure may also need suitable GPU hosts within the same supported network topology, which can make allocation harder.

At this point, I'd open an Azure Machine Learning compute provisioning support case and request a capacity and allocation investigation for the exact H100 v5 SKU in southafricanorth. Include the subscription ID, workspace resource ID, cluster name, requested node count, UTC timestamps, and any correlation or operation IDs.

Ask Microsoft to confirm whether the failure is caused by regional GPU capacity, cluster-specific placement constraints, or an Azure ML backend provisioning issue. If it's capacity-related, they should also be able to advise whether another region or SKU is the practical alternative.

Btw, what's the exact VM size you're requesting? Is it Standard_ND96isr_H100_v5? And does the same SKU fail for a one-node compute cluster every time, even after deleting the standalone compute instances? That would help narrow down whether we're dealing with cluster provisioning specifically or available GPU capacity.

rgds,

Alex

p.s. if my answer is help - pls mark it as an answer