Edit

Share via

Emit metrics for consumption of Azure OpenAI tokens

  • Article

APPLIES TO: All API Management tiers

The azure-openai-emit-token-metric policy sends custom metrics to Application Insights about consumption of large language model tokens through Azure OpenAI Service APIs. Token count metrics include: Total Tokens, Prompt Tokens, and Completion Tokens.

Note

Set the policy's elements and child elements in the order provided in the policy statement. Learn more about how to set or edit API Management policies.

Supported Azure OpenAI Service models

The policy is used with APIs added to API Management from the Azure OpenAI Service of the following types:

API type Supported models
Chat completion gpt-3.5

gpt-4
Completion gpt-3.5-turbo-instruct
Embeddings text-embedding-3-large

text-embedding-3-small

text-embedding-ada-002

For more information, see Azure OpenAI Service models.

Limits for custom metrics

Azure Monitor imposes usage limits for custom metrics that may affect your ability to emit metrics from API Management. For example, Azure Monitor currently sets a limit of 10 dimension keys per metric, and a limit of 50,000 total active time series per region in a subscription (within a 12 hour period).

These limits have the following implications for configuring custom metrics in an API Management policy such as emit-metric or azure-openai-emit-token-metric:

  • You can configure a maximum of 10 custom dimensions per policy.

  • The number of active time series generated by the policy within a 12 hour period is the product of the number of unique values of each configured dimension during the period. For example, if three custom dimensions were configured in the policy, and each dimension had 10 possible values within the period, the policy would contribute 1,000 (10 x 10 x 10) active time series.

  • If you configure the policy in multiple API Management instances that are in the same region in a subscription, all instances can contribute to the regional active time series limit.

Learn more about design limitations and considerations for custom metrics in Azure Monitor.

Prerequisites

Policy statement

<azure-openai-emit-token-metric
        namespace="metric namespace" >      
        <dimension name="dimension name" value="dimension value" />
        ...additional dimensions...
</azure-openai-emit-token-metric>

Attributes

Attribute Description Required Default value
namespace A string. Namespace of metric. Policy expressions aren't allowed. No API Management

Elements

Element Description Required
dimension Add one or more of these elements for each dimension included in the metric. Yes

dimension attributes

Attribute Description Required Default value
name A string or policy expression. Name of dimension. Yes N/A
value A string or policy expression. Value of dimension. Can only be omitted if name matches one of the default dimensions. If so, value is provided as per dimension name. No N/A

Default dimension names that may be used without value

  • API ID
  • Operation ID
  • Product ID
  • User ID
  • Subscription ID
  • Location
  • Gateway ID

Usage

Usage notes

  • This policy can be used multiple times per policy definition.
  • You can configure at most 10 custom dimensions for this policy.
  • This policy can optionally be configured when adding an API from the Azure OpenAI Service using the portal.
  • Where available, values in the usage section of the response from the Azure OpenAI Service API are used to determine token metrics.
  • Certain Azure OpenAI endpoints support streaming of responses. When stream is set to true in the API request to enable streaming, token metrics are estimated.

Example

The following example sends Azure OpenAI token count metrics to Application Insights along with API ID as a custom dimension.

<policies>
  <inbound>
      <azure-openai-emit-token-metric
            namespace="AzureOpenAI">   
            <dimension name="API ID" />
        </azure-openai-emit-token-metric> 
  </inbound>
  <outbound>
  </outbound>
</policies>

Related content

For more information about working with policies, see: