New to Azure AI Foundry - How should I validate deployment quota and throughput before production?

Mohamed, Rihan 40 Reputation points
2026-07-16T16:20:17.13+00:00

Hi everyone,

I'm new to Azure AI Foundry development and would appreciate some guidance.

I created an AI agent using the Azure AI Foundry SDK in Visual Studio Code and deployed it to Azure AI Foundry using a GPT-5.4 model. It is working as expected, but I have been assigned a task to validate whether the deployment has sufficient quota and throughput before production.

Since I'm new to this area, I'm not sure where to start.

Could someone help me understand:

  • What is the recommended approach for validating an Azure AI Foundry deployment?
  • How can I measure prompt and completion token usage?
  • How can I monitor Tokens Per Minute (TPM) and Requests Per Minute (RPM)?
  • How should I perform load testing (for example, simulating multiple concurrent users)?
  • What metrics should I monitor to determine whether the deployment can handle the expected workload?
  • How do I know when the current quota is insufficient and a quota increase is required?
  • Are there any Microsoft Learn articles, sample scripts, or best practices that demonstrate this process?

Any beginner-friendly guidance or documentation would be greatly appreciated.

Thank you!

Azure OpenAI in Foundry Models

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. kagiyama yutaka 4,915 Reputation points
    2026-07-22T07:42:18.83+00:00

    I think throughput is judged by running traffic that matches the real prompt size, completion size, and concurrency, and watching Foundry → Operate → Quota and Azure Monitor → Tokens / Requests / 429 during that load. Staying inside those limits without 429s or sustained high utilization shows the deployment can handle the expected workload.

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments
  2. Andrew Taylor - COREZENN 1,225 Reputation points Volunteer Moderator
    2026-07-27T19:43:19.5633333+00:00

    Hi Mohamed,

    Great question — quota and throughput validation is an important pre-production step, and the documentation covers it well. Here is a practical walkthrough based on the official guidance, which you referenced in your question.

    1. Understanding TPM and RPM

    When you deploy a model in Azure AI Foundry, quota is assigned in units of Tokens Per Minute (TPM). The platform automatically derives a corresponding Requests Per Minute (RPM) limit from that TPM value — you don't set them independently.

    2. View Your Current Quota Allocation

    In the Azure AI Foundry portal (ai.azure.com):

    1. Navigate to Management → Quota.
    2. You'll see quota allocated per model, per region, and how much is consumed across deployments.
    3. On the Deployments page, you can also view and adjust the TPM assigned to each individual deployment.

    This is the right starting point — before running any tests, confirm that the TPM you've assigned aligns with your expected production load.

    3. Monitor Token Usage and Rate Limits (Azure Monitor)

    Azure Monitor automatically collects metrics for your Azure OpenAI resource. Key metrics to watch:

    Metric What it tells you
    AzureOpenAITokenTransaction Total prompt + completion tokens consumed
    AzureOpenAITokenTransaction Total prompt + completion tokens consumed
    AzureOpenAITimeToResponse Latency from request to first byte (streaming)
    AzureOpenAITTLTInMS Time to last byte
    AzureOpenAITokenPerSecond Token generation speed
    AzureOpenAIProvisionedManagedUtilizationV2 PTU utilization % (for provisioned deployments)
    HTTP 429 response codes Indicates rate limit was hit

    To access these:

    1. Sign in to the Azure portal.
    2. Open your Azure OpenAI resource → Metrics.
    3. Select the metrics above and apply splitting by ModelDeploymentName to isolate your deployment.

    4. Monitor Your Agent Specifically (Agent Monitoring Dashboard)

    If you're using an AI Foundry agent (as you mentioned), there is a built-in monitoring dashboard:

    1. In Foundry portal, go to Build → select your agent.
    2. Select the Monitor tab.

    This dashboard shows:

    • Token usage over time
    • Latency per run
    • Run success rate (a rate below 95% warrants investigation)
    • Evaluation metrics

    High token usage may indicate verbose prompts that could be optimized.

    5. Load Testing (Simulating Concurrent Users)

    For simulating real-world traffic before production, the recommended approach is Azure Load Testing:

    It generates high-scale load against your endpoint.

    It integrates with Azure Monitor so you can observe token consumption and latency under stress.

    You can detect the point at which 429 throttling begins.

    Reference: Azure Load Testing overview

    A practical pattern from the quota docs is:

    Ramp up traffic gradually when onboarding new workloads. Avoid sharp spikes — RPM rate limits expect requests to be evenly distributed over each minute, and a burst within a 1–10 second window can trigger a 429 even if total requests are below the per-minute limit.

    6. Knowing When to Request a Quota Increase

    You likely need a quota increase when:

    • You are consistently hitting 429 responses near the end of a minute window.
    • Azure Monitor shows TPM/RPM utilization close to 100%.
    • Your projected production throughput (from load testing) exceeds the currently assigned TPM.

    To request an increase:

    Note from the docs: priority is given to customers who are generating traffic that already consumes their existing quota, so running real load tests before requesting helps.

    1. Check current quota: Management → Quota in Foundry portal.
    2. Assign appropriate TPM: Adjust on the Deployments page.
    3. Run a load test: Use Azure Load Testing to simulate production traffic.
    4. Watch Azure Monitor: Look for 429s, high latency, TPM consumption.
    5. Review agent metrics: Use the Agent Monitoring Dashboard (Monitor tab).
    6. Request a quota increase if 429s appear at expected production load.

    Please 'Upvote' (Thumbs-up) and 'Accept' as answer if the response was helpful. This will benefit other community members who face the same issue.

    Best regards,

    Andrew S Taylor

    Was this answer helpful?

    0 comments
  3. Anshika Varshney 15,280 Reputation points Microsoft External Staff Moderator
    2026-07-19T20:49:38.39+00:00

    Hello @Mohamed, Rihan ,

    Thanks for your question. A good approach is to validate capacity using realistic workload testing before moving to production.

    • Review your deployment quota in Azure AI Foundry > Operate > Quota.
    • Track actual usage during testing, including Tokens Per Minute (TPM), Requests Per Minute (RPM), prompt tokens, and completion tokens.
    • Use Azure Monitor metrics to monitor utilization, latency, errors, and throttling events (429 responses).
    • Perform load testing with realistic prompt sizes, response sizes, and expected concurrent users. Gradually increase traffic and observe system behavior.
    • Signs that additional quota may be required include frequent 429 errors, sustained high utilization, increased latency, or inability to meet expected throughput under normal workloads.

    For production readiness, monitor:

    • TPM and RPM consumption
    • Input/output token usage
    • Latency and error rates
    • 429 throttling responses
    • Deployment utilization trends

    Microsoft also provides guidance on quota management, provisioned throughput, and capacity planning in Azure AI Foundry Models documentation. Using those resources together with Azure Monitor metrics and realistic load testing will help determine whether your deployment can support the expected production workload.

    Hope this helps. Do let me know if you have any further queries.

    If this answers your query, please do click Accept Answer and Yes for was this answer helpful.

    Thankyou!

    Was this answer helpful?

  4. AI answer

    2026-07-16T16:20:58.7733333+00:00

    A practical validation flow is:

    1. Confirm quota allocation for the deployment
      • In Microsoft Foundry with New Foundry enabled, go to Operate > Quota.
      • Use the Token per minute tab for standard deployments and the Provisioned throughput unit tab for provisioned deployments.
      • Open the deployment details pane to see the current quota allocation, usage, and affiliated deployments using shared quota.
      • If needed, edit quota allocation from the deployment details pane to rebalance unused quota from other deployments.
      • After quota edits or requests, allow up to 15 minutes for changes to propagate.
    2. Estimate expected throughput before load testing
      • For provisioned deployments, throughput depends on PTUs deployed, input size, output size, and call rate.
      • Use the capacity calculator in the Foundry portal under the Quota page and Provisioned tab for an initial sizing estimate.
      • Then benchmark with a workload that matches real traffic.
    3. Measure actual utilization during testing
      • For provisioned deployments, use Azure Monitor on the Azure OpenAI resource:
        • Go to portal.azure.com > Azure OpenAI resource > Metrics.
        • Select Provisioned-managed utilization V2.
        • If there are multiple deployments, use Apply Splitting to view each deployment separately.
      • A deployment returns 429 on new calls after utilization rises above 100%.
    4. Monitor rate-limit behavior for standard deployments
      • Check per-deployment TPM allocation, not only subscription-level quota.
      • A deployment can return 429 even when subscription quota exists if the specific deployment does not have enough allocated TPM.
      • Use Azure Monitor metrics to review 24-hour and seven-day usage trends and identify bursty traffic patterns.
      • Rebalance quota across deployments based on observed usage.
    5. Run load testing in a controlled way
      • Benchmark using traffic that reflects the real workload.
      • Simulate the expected request rate, prompt size, and response size.
      • Increase traffic gradually instead of sending sharp bursts.
      • Spread requests across multiple deployments or regions if one deployment is not enough.
      • If immediate responses are not required, queue requests and process them at a controlled rate.
    6. Watch for signs that quota is insufficient
      • Repeated 429 responses during realistic traffic are a primary signal.
      • For provisioned deployments, utilization approaching or exceeding 100% indicates the endpoint is at capacity.
      • For standard deployments, quota may be insufficient when the deployment’s TPM allocation is too low for observed traffic, even if the subscription still has approved quota elsewhere.
      • If the model shows Insufficient quota and no quota row is available to adjust, submit a manual quota increase request.
    7. When to request more quota
      • Request more quota when realistic benchmark traffic cannot be handled after:
        • rebalancing TPM across deployments,
        • reducing burstiness,
        • or trying fewer PTUs for provisioned capacity constraints.
      • Quota increase requests are submitted through the quota request form. Requests are processed in order received, with priority given to customers already consuming existing quota.
    8. What to monitor for production readiness
      • Current deployment quota allocation
      • Usage trends over 24 hours and 7 days
      • 429 frequency during test runs
      • Provisioned utilization percentage for provisioned deployments
      • Request shape factors: requests per minute, average prompt size, average response size
      • Whether traffic is evenly distributed or bursty

    On token usage specifically:

    • The available guidance here distinguishes input size and output size as key throughput drivers.
    • TPM allocation is not related to a model’s max input token limit.
    • For PTU sizing, request shape is based on RPM, average prompt size (input tokens), and average response size (output tokens).

    A beginner-friendly approach is:

    1. Check the deployment’s current quota in Operate > Quota.
    2. Estimate capacity with the capacity calculator.
    3. Run a benchmark using realistic prompts and expected concurrency.
    4. Monitor Azure Monitor metrics during the test.
    5. If 429s appear or utilization reaches capacity, rebalance quota or request more.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.