Alert signals for open ai

Question

Alert signals for open ai

Andrew J 20

Hi everyone

I have open ai account setup in cognitive services, then I have created a deployment for gpt-4 to use it later, since I am not sure what load I would get there, I would like to set token limit value to be low. Then I want to set up an alert for 429 status code to be aware when throttling happens in order to increase tokens limit. I found 1 metric which is responsible for that: BlockedCalls in cognitiveservices namespace. It works fine for me: I have alert set up and tried to throttle api with requests and alert was triggered just fine. The only part I do not understand is the description of the signal itself. It says: do not use for azure open ai service. What exactly does that mean? since it works for me. And if I should not use it, whats the better way to alert on throttling?

Thanks for your help in advance!

Andrew J 20 Reputation points

2024-01-10T15:43:35.36+00:00

Found out that the reason is probably:

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring
These are legacy metrics that are common to all Azure AI Services resources. We no longer recommend that you use these metrics with Azure OpenAI.

Then the only way to monitor it is to write logs and then use custom metric which will search logs for 429 responses and will trigger the alert?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-01-11T10:09:35.0233333+00:00

Andrew J Greetings! Yes, It is recommended to monitor throttling for Azure OpenAI is to write logs and then use custom metrics to search for 429 responses and trigger alerts. You can use Azure Monitor to collect logs from your Azure OpenAI service and then create custom metrics based on those logs. Do let me know if you have any further queries.

Accepted answer

0 additional answers

Your answer

Andrew J 20 Reputation points

2024-01-10T15:43:35.36+00:00

Found out that the reason is probably:

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring
These are legacy metrics that are common to all Azure AI Services resources. We no longer recommend that you use these metrics with Azure OpenAI.

Then the only way to monitor it is to write logs and then use custom metric which will search logs for 429 responses and will trigger the alert?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-01-11T10:09:35.0233333+00:00

Andrew J Greetings! Yes, It is recommended to monitor throttling for Azure OpenAI is to write logs and then use custom metrics to search for 429 responses and trigger alerts. You can use Azure Monitor to collect logs from your Azure OpenAI service and then create custom metrics based on those logs. Do let me know if you have any further queries.

Answer 1

JananiRamesh-MSFT 29,261

@Andrew J Thanks for reaching out. Yes, that's correct. The recommended way to monitor throttling for Azure OpenAI services is to use custom logging and metrics. You can configure your Azure OpenAI service to log all requests and responses, including any 429 responses due to throttling. You can then use Azure Monitor to create a custom metric that tracks the number of 429 responses over a given time period. You can configure an alert rule to trigger when the number of 429 responses exceeds a certain threshold. Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring#configure-diagnostic-settings https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring#set-up-alerts

do let me know incase of further queries, I would be happy to assist you.

Andrew J 20 Reputation points

2024-01-11T14:22:09.9766667+00:00

Indeed, to monitor 429 responses one needs to set up logging and then create custom metric. It can be used to fire an alert. Thanks for your time! @JananiRamesh-MSFT and @AshokPeddakotla-MSFT
JananiRamesh-MSFT 29,261 Reputation points

2024-01-11T14:59:33.9766667+00:00

Andrew J Appreciate you sharing the solution with the community, as per Microsoft Q&A community policy: "The question author cannot accept their own answer. They can only accept answers by others", I have converted my comment to answer. Feel free to 'Accept as Answer' as this will help increase the visibility of this question and the solution for anyone else who may run into the same issue. Thank you for helping to improve Microsoft Q&A!

Share via

Alert signals for open ai

0 additional answers

Your answer