Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,277 questions
Does the Azure OpenAI dynamic quota for deployments feature allows a deployment to use TPMs quotas from other deployments for the same mode/region?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 69%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Johannes Borch
0
Reputation points
I have a setup in Azure OpenAI where I have two deployments using the same model/region/subscription sharing the model TPM quota equally (50%). If I enable the new dynamic quota feature on my deployments, will one model be able to consume the full model quota at 100% when the other deployment does not consume tokens? Or is the dynamic quota feature just a way to increase the regional model quota temporarily if Azure region has capacity?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 1%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYM%3C/text%3E%3C/svg%3E)
YutongTie-MSFT 53,211 Reputation points2024-07-12T01:26:14.74+00:00 Hello Johannes,
Thanks for reaching out to us and welcome to Q&A community. As the document -
Dynamic quota allows a deployment to take advantage of available capacity in service. If capacity is available, we will dynamically increase your quota and receive higher throughput.
An example here, when you are not in heavily utilized regions, and avoiding peak hours of use for those regions, you may get higher quota according to the capacity. If you assigned 200K to the deployment, then your throughput will be 200K if there is no capacity, and will be higher when there is capacity. It should be not seen as move your potential quota from your other deployment, but from the whole region.
Dynamic quota primarily adjusts the total available tokens in the region based on overall demand and capacity, potentially increasing the available quota for all deployments collectively.
I hope this helps! Let us know if you need more information.
Regards,
Yutong
-
Johannes Borch 0 Reputation points
2024-07-12T05:59:04.1566667+00:00 Hi Yutong, thanks for the explanation and it confirms my understanding that the dynamic quota feature adjusts the model token quota in a region based on available capacity.
So if I extend your example to two deployments at 100K for a model with quota of 200K in a given region. (I can't assign more to the deployment because the sum of rate limit needs to < model quota). Will a single deployment be able to burst to 200K TPM or even above if region has capacity? Or will it be capped at 100 TPM?Best, Johannes
Sign in to comment
Sign in to answer