Loging ModelNames in API mangement for Background requests

Question

Hello.

Im using the llm-emit-token-metric policy to log token metric so that we can distribute cost across projecst in our company.

This works fine for most request. there are however. When you do a background request to a gpt model by setting background to true in the body. It starts proccessing the response in the background and you can latter get it using a get and a response_id. whe it completes the llm-emit-token-metric logs the tokens. however on the get we only know the model when we get ther response back since for get calls you only send the response id and not the model name. And the llm-emit-token-metric can only be put in the inbound section. So is there a way to ad the modelname as custum dimension for llm-emit-token-metric when using background requests?

best of regards,

Jeffrey Bakker

Answer 1

Hello Jeffrey,

Thank you for reaching out on Microsoft Q&A portal.

As you noted, background polling operations via GET requests typically do not return or accept the model name, making it impossible for the metric policy to automatically detect it on the follow-up call. The most reliable workaround is to cache the model name during the initial POST request and retrieve it during the GET request so it can be injected into your metrics.

According to the official Microsoft documentation, the llm-emit-token-metric policy "sends custom metrics to Application Insights about consumption of large language model (LLM) tokens through LLM APIs." You can "configure at most 5 custom dimensions for this policy," and it is strictly evaluated in the <inbound> section. Furthermore, the metrics capture process dictates that "Values in the usage section of the response from the LLM API, when available, are used to determine token metrics."

Here is how you can securely structure your APIM policies using cache-store-value and cache-lookup-value while ensuring your HTTP streams are preserved.

1. Initial POST Request (Store Model)

Extract the requested model from the inbound request and the response_id from the outbound response, using preserveContent: true to prevent the stream from being consumed.

<policies>
    <inbound>
        <base />
        <set-variable name="requestModel" value="@(context.Request.Body.As<JObject>(preserveContent: true)["model"]?.ToString())" />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
        <set-variable name="responseId" value="@(context.Response.Body.As<JObject>(preserveContent: true)["response_id"]?.ToString())" />
        
        <choose>
            <when condition="@(!string.IsNullOrEmpty((string)context.Variables["responseId"]) && !string.IsNullOrEmpty((string)context.Variables["requestModel"]))">
                <cache-store-value 
                    key="@("model_" + (string)context.Variables["responseId"])" 
                    value="@((string)context.Variables["requestModel"])" 
                    duration="300" />
            </when>
        </choose>
    </outbound>
</policies>

2. Background GET Request (Retrieve Model & Emit Metric)

Extract the response_id from the URL parameters (assuming it is passed in the path) and emit the custom dimension prior to sending the GET call to the backend.

<policies>
    <inbound>
        <base />
        <set-variable name="pollingResponseId" value="@(context.Request.MatchedParameters["response_id"])" />
        
        <cache-lookup-value 
            key="@("model_" + (string)context.Variables["pollingResponseId"])" 
            variable-name="cachedModel" />
            
        <choose>
            <when condition="@(context.Variables.ContainsKey("cachedModel"))">
                <llm-emit-token-metric namespace="MyLLMMetrics">
                    <dimension name="ModelName" value="@((string)context.Variables["cachedModel"])" />
                </llm-emit-token-metric>
            </when>
            <otherwise>
                <llm-emit-token-metric namespace="MyLLMMetrics" />
            </otherwise>
        </choose>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
</policies>

Please try and let me know if it works.

Note: This response is generated with the help of AI systems.

Answer 2

Hello Bakker, J. (Jeffrey),

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your Login ModelNames in API mangement for Background requests is having isue.

This issue fix is only by correlating the background POST and GET. The model name cannot be reliably added from the GET response because llm-emit-token-metric runs in inbound policy. Use an external Redis-compatible cache or durable correlation store, store response_id > model during the POST, retrieve it during the GET, and pass it as the ModelName dimension. Avoid adding response_id permanently as a metric dimension because it is high-cardinality and can hit Azure Monitor custom metric limits.

Use the below resource links for more reading and implementation steps:

• Azure API Management llm-emit-token-metric policy
• Azure API Management cache-store-value policy
• Azure API Management cache-lookup-value policy
• Azure API Management policy expressions
• Integrate Azure API Management with Application Insights

---

Please do not close the thread by upvoting and accepting the answer if any part of it is helpful.