Seeking Strategies to Enable SSE While Preserving Logs in Azure API Management

Question

Hello, I am currently utilizing Azure OpenAI Service (AOAI) as a backend, while managing authentication and logs with API Management Service (APIM). Logs are consolidated into Log Analytics Workspace and summarized in Application Insights. The size of both requests and responses from AOAI is large, exceeding the 8192 bytes limit from APIM to Application Insights. As a workaround, I parse the response to extract only the usage token amount and save it in Application Insights using emit-metrics.

To enable asynchronous communication with AOAI, I followed the guidance in this article to retrieve policy expressions: https://learn.microsoft.com/en-us/azure/api-management/how-to-server-sent-events The document mentions that to activate SSE, it's not possible to store Body logs in either Azure Monitor or Application Insights. Indeed, SSE does not become active when emit-metrics is specified in the outbound policy.

Given the need to audit AOAI usage and verify statistical amounts, my goal is to log the contents documented in the Response Body. What approach could allow for the activation of SSE while still enabling the logging of the Body?

Answer 1

@SS Unfortunately, with just APIM, I don't believe there is a solution to tackle your scenario. Instead, here are two options you could consider

1. Client-Side Logic
   If you control the client application (web app, SPA, native, etc.), you could leverage client-side telemetry collection to report back all the data you need once the response is completely streamed to the client. This would be the simpler option if applicable to you.
2. Proxy Service
   This approach involves building a proxy-service between APIM and Azure OpenAI that can proxy the SSE request while looking into the chunks for information and recording it. This is a bit more complex but would account for all client applications that use your API, especially if you do not control all client applications.