Emit metrics for consumption of Azure OpenAI tokens
APPLIES TO: All API Management tiers
The azure-openai-emit-token-metric
policy sends metrics to Application Insights about consumption of large language model tokens through Azure OpenAI Service APIs. Token count metrics include: Total Tokens, Prompt Tokens, and Completion Tokens.
Note
Set the policy's elements and child elements in the order provided in the policy statement. Learn more about how to set or edit API Management policies.
Supported Azure OpenAI Service models
The policy is used with APIs added to API Management from the Azure OpenAI Service of the following types:
API type | Supported models |
---|---|
Chat completion | gpt-3.5 gpt-4 |
Completion | gpt-3.5-turbo-instruct |
Embeddings | text-embedding-3-large text-embedding-3-small text-embedding-ada-002 |
For more information, see Azure OpenAI Service models.
Prerequisites
- One or more Azure OpenAI Service APIs must be added to your API Management instance. For more information, see Add an Azure OpenAI Service API to Azure API Management.
- Your API Management instance must be integrated with Application insights. For more information, see How to integrate Azure API Management with Azure Application Insights.
- Enable Application Insights logging for your Azure OpenAI APIs.
- Enable custom metrics with dimensions in Application Insights. For more information, see Emit custom metrics.
Policy statement
<azure-openai-emit-token-metric
namespace="metric namespace" >
<dimension name="dimension name" value="dimension value" />
...additional dimensions...
</azure-openai-emit-token-metric>
Attributes
Attribute | Description | Required | Default value |
---|---|---|---|
namespace | A string. Namespace of metric. Policy expressions aren't allowed. | No | API Management |
value | Value of metric expressed as a double. Policy expressions are allowed. | No | 1 |
Elements
Element | Description | Required |
---|---|---|
dimension | Add one or more of these elements for each dimension included in the metric. | Yes |
dimension attributes
Attribute | Description | Required | Default value |
---|---|---|---|
name | A string or policy expression. Name of dimension. | Yes | N/A |
value | A string or policy expression. Value of dimension. Can only be omitted if name matches one of the default dimensions. If so, value is provided as per dimension name. |
No | N/A |
Default dimension names that may be used without value
- API ID
- Operation ID
- Product ID
- User ID
- Subscription ID
- Location
- Gateway ID
Usage
- Policy sections: inbound
- Policy scopes: global, workspace, product, API, operation
- Gateways: classic, v2, consumption, self-hosted, workspace
Usage notes
- This policy can be used multiple times per policy definition.
- You can configure at most 10 custom dimensions for this policy.
- This policy can optionally be configured when adding an API from the Azure OpenAI Service using the portal.
- Where available, values in the usage section of the response from the Azure OpenAI Service API are used to determine token metrics.
- Certain Azure OpenAI endpoints support streaming of responses. When
stream
is set totrue
in the API request to enable streaming, token metrics are estimated.
Example
The following example sends Azure OpenAI token count metrics to Application Insights along with User ID, Client IP, and API ID as dimensions.
<policies>
<inbound>
<azure-openai-emit-token-metric
namespace="AzureOpenAI">
<dimension name="User ID" />
<dimension name="Client IP" value="@(context.Request.IpAddress)" />
<dimension name="API ID" />
</azure-openai-emit-token-metric>
</inbound>
<outbound>
</outbound>
</policies>
Related policies
- Logging
- emit-metric policy
- azure-openai-token-limit policy
Related content
For more information about working with policies, see:
- Tutorial: Transform and protect your API
- Policy reference for a full list of policy statements and their settings
- Policy expressions
- Set or edit policies
- Reuse policy configurations
- Policy snippets repo
- Author policies using Microsoft Copilot in Azure