Getting 429 Too many requests when using Azure Cost Management API in Grafana Dashboard

Question

Getting 429 Too many requests when using Azure Cost Management API in Grafana Dashboard

Priya Sukumaran (Adfolks LLC) 0

While using the Azure Cost Management API in the Grafana dashboard, we are encountering a 429 Too Many Requests error. We've implemented the suggested solutions from previous inquiries, including adding the ("ClientType": "Dashboard") header and the x-ms-command-name header as CostAnalysis, but we are still experiencing the 429 error. Please advise on how to resolve this issue.

SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-10-14T16:45:14.6233333+00:00

@Priya Sukumaran (Adfolks LLC) Thank you for reaching out to us.

We have this rate limit in place to ensure that the service reliability is not impacted with too many requests coming in at a short time.

I understand that you have gone through the suggestions for your issue from earlier discussions. Appreciate if you could share what all suggestions you followed and also provide more details about your set up: Number of tenants, subscriptions and the request that you are making (API call details).

Sharing similar discussions, might be helpful:
https://learn.microsoft.com/en-us/answers/questions/1340993/exception-429-too-many-requests-for-azure-cost-man

https://learn.microsoft.com/en-us/answers/questions/1433407/azure-cost-management-api-returning-429-too-many-r

https://learn.microsoft.com/en-us/answers/questions/1603465/429-too-many-requests-when-using-azure-cost-manage

https://learn.microsoft.com/en-us/answers/questions/1488793/costmanagement-query-operation-too-many-request-on
Priya Sukumaran (Adfolks LLC) 0 Reputation points

2024-10-24T05:35:58.1633333+00:00

Hi @SadiqhAhmed-MSFT ,

We are using the community version of grafana for monitoring purpose. Currently we are retriving data for one subsciption inclusing annual usage Cost per Servies and comparing bugget against cost .This results in more that 4 API calls from a single dashboard.

We are displaying this data in a playlist

We have added ("ClientType": "Dashboard") header and the x-ms-command-name header as CostAnalysis.
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-10-25T21:08:39.3+00:00
@Priya Sukumaran (Adfolks LLC) Sorry for the delay!

In order to avoid any 429 errors for the Microsoft.CostManagement/Query endpoint API, below is our recommendations:

Add 20 seconds wait time between each API call.

For example: first call is UTC 8:00:00AM, the 2nd call would be 8:00:20AM, and the 3rd call would be 8:00:40AM.

For the next call, it would be 8:01:00AM, where the rate limit has already been refreshed for that minute.

Even we don't count for the API response time (we just assume we call the API and the API returns the response in 0 seconds.), in each minute, you only can call the API 3 times most.

In that way, you can avoid entity, tenant and QPU rate limit. (QPU might have more dimensions to calculate but mostly you won't hit qpu rate limit if you follow this rule.)

Add a ClientType request header.

The client request header is a http header that you can send along with the Authorization header.

The key would be ClientType, the value could be any string, we usually recommend customer to use their company name, customer name, or team name.

Hope this helps!

1 answer

Your answer

SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-10-14T16:45:14.6233333+00:00

@Priya Sukumaran (Adfolks LLC) Thank you for reaching out to us.

We have this rate limit in place to ensure that the service reliability is not impacted with too many requests coming in at a short time.

I understand that you have gone through the suggestions for your issue from earlier discussions. Appreciate if you could share what all suggestions you followed and also provide more details about your set up: Number of tenants, subscriptions and the request that you are making (API call details).

Sharing similar discussions, might be helpful:
https://learn.microsoft.com/en-us/answers/questions/1340993/exception-429-too-many-requests-for-azure-cost-man

https://learn.microsoft.com/en-us/answers/questions/1433407/azure-cost-management-api-returning-429-too-many-r

https://learn.microsoft.com/en-us/answers/questions/1603465/429-too-many-requests-when-using-azure-cost-manage

https://learn.microsoft.com/en-us/answers/questions/1488793/costmanagement-query-operation-too-many-request-on
Priya Sukumaran (Adfolks LLC) 0 Reputation points

2024-10-24T05:35:58.1633333+00:00

Hi @SadiqhAhmed-MSFT ,

We are using the community version of grafana for monitoring purpose. Currently we are retriving data for one subsciption inclusing annual usage Cost per Servies and comparing bugget against cost .This results in more that 4 API calls from a single dashboard.

We are displaying this data in a playlist

We have added ("ClientType": "Dashboard") header and the x-ms-command-name header as CostAnalysis.
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-10-25T21:08:39.3+00:00

@Priya Sukumaran (Adfolks LLC) Sorry for the delay!

In order to avoid any 429 errors for the Microsoft.CostManagement/Query endpoint API, below is our recommendations:

Add 20 seconds wait time between each API call.

For example: first call is UTC 8:00:00AM, the 2nd call would be 8:00:20AM, and the 3rd call would be 8:00:40AM.

For the next call, it would be 8:01:00AM, where the rate limit has already been refreshed for that minute.

Even we don't count for the API response time (we just assume we call the API and the API returns the response in 0 seconds.), in each minute, you only can call the API 3 times most.

In that way, you can avoid entity, tenant and QPU rate limit. (QPU might have more dimensions to calculate but mostly you won't hit qpu rate limit if you follow this rule.)

Add a ClientType request header.

The client request header is a http header that you can send along with the Authorization header.

The key would be ClientType, the value could be any string, we usually recommend customer to use their company name, customer name, or team name.

Hope this helps!

Answer 1

Hi Priya Sukumaran (Adfolks LLC),

Thanks for reaching out to Microsoft Q&A.

The "429 Too Many Requests" error typically indicates that you might have exceded the API's rate limit. Even after adding the recommended headers, it is possible that the volume or freq of requests is still too high.

Here are some possible steps you can take to mitigate this issue...

Check API Limits: Review the official rate limits for the Azure Cost Management API. Ensure that your dashboard doesn't exceed the maximum number of requests allowed per minute/hour.

Implement Retry Logic: Ensure your application has builtin retry logic with exponential backoff to handle temp rate limit errors. This will automatically reduce the load on the API when rate limits are hit
Cache Results: Instead of fetching data in realtime for every request, consider caching API results for a certain period (ex: 5 minutes). This reduces the number of API calls needed, especially for a dashboard that may refresh frequently.
Reduce Frequency of API Calls: Modify the refresh rate of your grafana dashboard. If it is refreshing too often, you may be hitting the rate limits quicker than expected.

Use Cost Management Exports: Instead of querying the API frequently, consider using Azure Cost Management’s "Cost Export" feature to export cost data to a storage account and query the data from there, reducing the need for frequent API calls.

Monitor API Usage: Track the number of API requests your dashboard is making. Azure provides built-in metrics and logging to monitor API usage, which can help you identify patterns and optimize calls.

These steps should help reduce the 429 errors by lowering the request frequency or handling rate limits more gracefully. Let me know if you need further guidance on implementing any of these.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Priya Sukumaran (Adfolks LLC) 0 Reputation points

2024-10-24T05:39:14.7766667+00:00

Hi @Vinodh247

Thank you for your response.

Will try these options.

Share via

Getting 429 Too many requests when using Azure Cost Management API in Grafana Dashboard

1 answer

Your answer