Custom metric to track token usage based on metadata in payload?

Question

Hi,

We're using Azure OpenAI deployments to generate completions. I want to track the token usage (=cost) of the requests we make in the Azure Monitoring dashboard. We make requests for several different purposes, let's say that each purpose has a UUID which could be exposed to the Azure OpenAI deployment as metadata somehow when we request a completion.

I want to split or filter on these UUIDs. Is this possible? I've looked around extensively but haven't found anything relevant yet.

Answer 1

Hi fredrikfridborn,

Thanks for reaching out to Microsoft Q&A.

To track token usage in Azure OpenAI deployments and filter by a custom UUID for each request, you can consider the following approach:

Embedding Metadata in Requests: When you send a request to the Azure OpenAI endpoint, include the UUID in a way that you can track it later, like within the request headers or payload. Although Azure OpenAI doesn’t directly support custom metadata, you can log this in your application layer.

Custom Logging and Metrics Collection:

• In your application that interacts with Azure OpenAI, you can log the UUID along with the token usage for each completion request. Azure OpenAI responses contain token usage information (prompt tokens, completion tokens, and total tokens).
  • Use Azure Monitor Logs or Azure Application Insights to track these custom logs. You can send custom logs from your application to Application Insights, including both the UUID and token usage.

1. Token Usage Monitoring:
   • After receiving the response from Azure OpenAI, extract the token usage data from the response ('usage' object in the API response).
   • Combine the UUID and token usage in a custom metric and send it to Azure Monitor using Azure Monitor Custom Metrics or Application Insights.
   • This will allow you to filter and split the token usage based on the UUID in your monitoring dashboard.

Visualizing in Azure Monitor:

• Use Azure Metrics Explorer or Azure Log Analytics to create visualizations based on the custom metric you’ve sent (e.g., token usage per UUID). You can filter and split by UUID in your dashboards.

Here is the general flow:

1. Send request with UUID.
2. Receive token usage in response.
3. Log UUID and token usage.
4. Push custom metrics to Azure Monitor/Application Insights.
5. Create filters and visualizations based on UUID.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.