Model Deployment Stuck at 0 Requests per Minute

Question

Model Deployment Stuck at 0 Requests per Minute

Leo Grant 5

Hi,

My model deployment is stuck at 0 RPM despite having 1000 tokens per minute rate limit.

How can I change this as it seems the RPM is calculated based on the TKM but this isn't changing for me?

Thanks for any help you can offer!

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-08-26T07:37:06.41+00:00
Leo Grant Greetings & Welcome to Microsoft Q&A forum!

What is the model name and the region?

You can check the status of your deployment under Deployments tab of Azure OpenAI Studio.

As mentioned here, TPM rate limits are based on the maximum number of tokens that are estimated to be processed by a request at the time the request is received. It isn't the same as the token count used for billing, which is computed after all processing is completed.

As each request is received, Azure OpenAI computes an estimated max processed-token count that includes the following:

Prompt text and count

The max_tokens parameter setting

The best_of parameter setting

As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets.

How can I change this as it seems the RPM is calculated based on the TKM but this isn't changing for me?

First check if you have reached the quota from Azure OpenAI studio.

To increase the quota, In Azure OpenAI studio, under shared resources -> select quota.

Check if the quota is already full or not.

You choose Request quota link to increase the current model quota if already full.

Also, See Manage Azure OpenAI Service quota for more details.

Do let me know if that helps or have any further queries.
Leo Grant 5 Reputation points

2024-08-26T10:51:16.1066667+00:00

Thanks for helping out!

So I've tried deploying gpt-4 model version turbo-2024-04-09 using the Global-Standard deployment into East-US2.

However, it seems that any model I create has 0 RPM (see screenshot below) implicitly assigned.

I've attached the quota below, and also attached what I see when I try to edit the rate limit of my model (which is why I'm confused I get 0RPM instead of 6RPM).

Maybe I'm missing something?

Just to add also, this is how I'm checking that my model has not used any tokens:
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-02T04:12:36.5066667+00:00

Leo Grant Apologies for the delayed response. Are you still seeing this issue?

To your follow-up query, The token count used in the rate limit calculation is an estimate based in part on the character count of the API request.

The rate limit token estimate is not the same as the token calculation that is used for billing/determining that a request is below a model's input token limit. Due to the approximate nature of the rate limit token calculation, it is expected behavior that a rate limit can be triggered prior to what might be expected in comparison to an exact token count measurement for each request.

Since you are only seeing 0 tokens all the time, to isolate ethe issue with the model, could you try with any other available model and see if that solves the issue?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-04T15:19:32.37+00:00

Leo Grant Do let me know if you are still blocked.

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-08-26T07:37:06.41+00:00

Leo Grant Greetings & Welcome to Microsoft Q&A forum!

What is the model name and the region?

You can check the status of your deployment under Deployments tab of Azure OpenAI Studio.

As mentioned here, TPM rate limits are based on the maximum number of tokens that are estimated to be processed by a request at the time the request is received. It isn't the same as the token count used for billing, which is computed after all processing is completed.

As each request is received, Azure OpenAI computes an estimated max processed-token count that includes the following:

Prompt text and count

The max_tokens parameter setting

The best_of parameter setting

As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets.

How can I change this as it seems the RPM is calculated based on the TKM but this isn't changing for me?

First check if you have reached the quota from Azure OpenAI studio.

To increase the quota, In Azure OpenAI studio, under shared resources -> select quota.

Check if the quota is already full or not.

You choose Request quota link to increase the current model quota if already full.

Also, See Manage Azure OpenAI Service quota for more details.

Do let me know if that helps or have any further queries.
Leo Grant 5 Reputation points

2024-08-26T10:51:16.1066667+00:00

Thanks for helping out!

So I've tried deploying gpt-4 model version turbo-2024-04-09 using the Global-Standard deployment into East-US2.

However, it seems that any model I create has 0 RPM (see screenshot below) implicitly assigned.

I've attached the quota below, and also attached what I see when I try to edit the rate limit of my model (which is why I'm confused I get 0RPM instead of 6RPM).

Maybe I'm missing something?

Just to add also, this is how I'm checking that my model has not used any tokens:
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-02T04:12:36.5066667+00:00

Leo Grant Apologies for the delayed response. Are you still seeing this issue?

To your follow-up query, The token count used in the rate limit calculation is an estimate based in part on the character count of the API request.

The rate limit token estimate is not the same as the token calculation that is used for billing/determining that a request is below a model's input token limit. Due to the approximate nature of the rate limit token calculation, it is expected behavior that a rate limit can be triggered prior to what might be expected in comparison to an exact token count measurement for each request.

Since you are only seeing 0 tokens all the time, to isolate ethe issue with the model, could you try with any other available model and see if that solves the issue?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-04T15:19:32.37+00:00

Leo Grant Do let me know if you are still blocked.

Share via

Model Deployment Stuck at 0 Requests per Minute

Your answer