I have created a Azure ML pipeline using Azure Python SDK and a compute cluster. When I submit the pipeline with compute cluster having maximum of 5 nodes, it always uses 1 node. Could someone explain why this happens and what can I do to use all 5 nodes?

Jhanvi 0

User's image

User's image

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-09-15T08:37:13.36+00:00

@Jhanvi Is there a resource lock on your resource group that contains your workspace? This prevents scaling operations for Azure Machine Learning compute clusters.

Also, if the run does not require additional resources scaling will not occur as the cluster is automatically scaling nodes up or down based on the number of runs submitted, and the max_nodes set on your cluster. If the runs are not high and if a single run does not require more than one node, it would not use all the 5 nodes. See this page for limitations of compute cluster. Thanks!!
Jhanvi Lathia 0 Reputation points

2023-09-15T09:21:09.16+00:00

@romungi-MSFT Thank you for your reply! Even if I have 2 pipelines using the same cluster, it does not scale up. It shares that single node.

Also, I checked about the locks, there is none.

Your answer