Why my quota are in use while I shut down all the machines in Azure ML?

Question

Why my quota are in use while I shut down all the machines in Azure ML?

Afshin Amiri 10

When deploying a model in Azure ML, I got the "not enough quota". So I shut down the machines in Azure ML but I get the same message again:

User's image

Then I checked the portal "Quota Usage" and it is like this:

available_shown

First question: Are these two conflicting? If not how do I am supposed to free up my usage ( and not request more quota)

Second Question: Even this is so, is there a workaround for this problem?

Thanks

kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T18:23:15.05+00:00

Hello, @Afshin Amiri !

There is a VM family vCPU quota as well as a VM regional vCPU quota so you may be running into a scenario where you can deploy VMs in a specific region (like West US) but not for a specific VM family (like the N-series GPU VMs).
Afshin Amiri 10 Reputation points

2023-03-02T18:49:55.29+00:00

I remember that I created these resources and (as you see from the first image) I have deleted those machines.How come they are still there and considered as my used quota? I only use Canada central region.
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T19:00:50.86+00:00

Let me take a look and I'll let you know what I run into. It may be that there's a delay or a quota measurement that isn't immediately obvious but I want to make sure that everything is working as intended.
Afshin Amiri 10 Reputation points

2023-03-02T19:22:07.5833333+00:00

thanks very much.

but look I understand the issue that you may have here. If this is the case that one UI is not updated and another is. I suggest you take some of your medicine and use "Storage Event Trigger" in "Azure Data Factory".
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T22:19:48.51+00:00
Hello, @Afshin Amiri !

Just a couple of quick follow-up questions:

In the first screenshot, you show a single workspace. Do you have any other workspaces if you go back to the Machine Learning Studio Workspaces tab (by clicking back on Zown on the screenshot) that may have compute resources?

In the second screenshot, it looks like you are close to your quota. Are you trying to deploy more than 3-4 vCPUs?
Afshin Amiri 10 Reputation points

2023-03-02T23:41:14.7266667+00:00

the point is that I had not been using resource at the time I took the screen shot.
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T23:49:54.4266667+00:00

Thank you for the clarification! I'm going to adjust the tags so that we can get a Machine Learning specialist to take a closer look at this.
Afshin Amiri 10 Reputation points

2023-03-02T23:50:06.49+00:00

Dear support Team,

I do not have other workspaces:

However,FYI, I have turned one machine on after posting those two images. which has been added up. And yet I could not find those other resources.
romungi-MSFT 29,791 Reputation points Microsoft Employee

2023-03-03T05:20:49.61+00:00

Are there any other compute instances in the other tabs of your compute page?

Are you also using any endpoints that could be using your quota? If any deployments are used an additional compute is reserved for upgrade scenarios and this may also use your quota.

Please check the documentation on this page on how the quotas are set at workspace and subscription level. This might help you determine if there are any additional resources that are running in your subscription.

I believe you have already deleted the instances rather than stopping them. Stopping the instances will not release the resources back to the quota.

If all the resources are still deleted but the quota does not reset, then there could be an issue with cleanup of the resources at the backend. You can request the service team to look into this by raising a Azure support case from your Azure portal without requesting an increase of your quota.

kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T18:23:15.05+00:00

Hello, @Afshin Amiri !

There is a VM family vCPU quota as well as a VM regional vCPU quota so you may be running into a scenario where you can deploy VMs in a specific region (like West US) but not for a specific VM family (like the N-series GPU VMs).
Afshin Amiri 10 Reputation points

2023-03-02T18:49:55.29+00:00

I remember that I created these resources and (as you see from the first image) I have deleted those machines.How come they are still there and considered as my used quota? I only use Canada central region.
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T19:00:50.86+00:00

Let me take a look and I'll let you know what I run into. It may be that there's a delay or a quota measurement that isn't immediately obvious but I want to make sure that everything is working as intended.
Afshin Amiri 10 Reputation points

2023-03-02T19:22:07.5833333+00:00

thanks very much.

but look I understand the issue that you may have here. If this is the case that one UI is not updated and another is. I suggest you take some of your medicine and use "Storage Event Trigger" in "Azure Data Factory".
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T22:19:48.51+00:00

Hello, @Afshin Amiri !

Just a couple of quick follow-up questions:

In the first screenshot, you show a single workspace. Do you have any other workspaces if you go back to the Machine Learning Studio Workspaces tab (by clicking back on Zown on the screenshot) that may have compute resources?

In the second screenshot, it looks like you are close to your quota. Are you trying to deploy more than 3-4 vCPUs?
Afshin Amiri 10 Reputation points

2023-03-02T23:41:14.7266667+00:00

the point is that I had not been using resource at the time I took the screen shot.
kobulloc-MSFT 13,586 Reputation points Microsoft Employee

2023-03-02T23:49:54.4266667+00:00

Thank you for the clarification! I'm going to adjust the tags so that we can get a Machine Learning specialist to take a closer look at this.
Afshin Amiri 10 Reputation points

2023-03-02T23:50:06.49+00:00

Dear support Team,

I do not have other workspaces:

However,FYI, I have turned one machine on after posting those two images. which has been added up. And yet I could not find those other resources.
romungi-MSFT 29,791 Reputation points Microsoft Employee

2023-03-03T05:20:49.61+00:00

Are there any other compute instances in the other tabs of your compute page?

Are you also using any endpoints that could be using your quota? If any deployments are used an additional compute is reserved for upgrade scenarios and this may also use your quota.

Please check the documentation on this page on how the quotas are set at workspace and subscription level. This might help you determine if there are any additional resources that are running in your subscription.

I believe you have already deleted the instances rather than stopping them. Stopping the instances will not release the resources back to the quota.

If all the resources are still deleted but the quota does not reset, then there could be an issue with cleanup of the resources at the backend. You can request the service team to look into this by raising a Azure support case from your Azure portal without requesting an increase of your quota.

Answer 1

Dear Support team

Thank you for your help and guiding me through different possibilities.

In fact, it was the endpoints which were consuming the resources.

I had 11 missing compute cores:

My colleague and I had created 3 one-core machines and one 4-core machine with 2 instances which add up to 11 cores. (Though the deployment failed but it consumed and dedicated the resource.

One I shut them down, the resource got free.

Note for fellow developers: At first, I was using real-time endpoint(quick) and then I realized it assign number of instances to more than 1 for load balancing. If you (like me) are in short of resources, do not use this option and go with real-time endpoint.

Thank you very much again.

Why my quota are in use while I shut down all the machines in Azure ML?

1 answer

Activity