How to track and monitor LLM model invocation on Azure OpenAI ?

Question

Hi everyone,

It seem easy question , but I struggle to look for the approach track and monitor LLM model invocation ,Previously, I worked on AWS Bedrock which is generative-ai platform on AWS , it provide Chat-Playground to test chat. It allow me to monitor model invocation using CloudWatch Logs as the link below. https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html

It help me simply track all messages conversation both input and output and total number of tokens as well. What services in Azure similar to what is provided in Amazon Bedrock ?

Let 's see log conversation messages below on AWS cloud watch that keep track of all conversation from Amazon bedrock.

Answer 1

Thanks for reaching out to us, it should be Azure Monitor service as comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.

The product page - https://azure.microsoft.com/en-us/products/monitor/

The document page - https://learn.microsoft.com/en-us/azure/azure-monitor/overview

The following diagram shows a high-level architecture view of Azure Monitor.

Please take a look at it and I hope it helps! If you have any question about Azure Monitor Service, please feel free to open a thread with the Azure Monitor tag.

Also, Azure OpenAI provides out-of-box dashboards for each of your Azure OpenAI resources. To access the monitoring dashboards sign-in to https://portal.azure.com and select the overview pane for one of your Azure OpenAI resources.

More information you may want to learn - https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitor-openai

Hope this helps!

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.