Investigating slow response time in Azure API Management and backend

Question

I have an API Management with the default Azure gateway that exposes an API deployed on-premise. The first request of the day takes a long time, more than 30 seconds. Reviewing the logs, I see that the Backend Time is almost equally slow, but nevertheless, the API responds in 2 seconds. It's not a production environment, so the call volume is very small. There are no policies configured for this api. I have checked the Allow trace option, enable Diagnostic settings and integrate the api and the ApiM with ApplicationInsights to review the e2e transaction information. How can I know where that time is lost? Is there any Azure service that i could use to improve the request trazability?

Answer 1

Jesus Arnas Iñigo Thanks for posting your question in Microsoft Q&A. I would like to start with diagnostic settings logs which has different properties that help in identifying where the slowness occurs. You can use the below query as example (for Azure Monitor):

ApiManagementGatewayLogs
| where TimeGenerated > ago(1d)
| extend ApimTime = TotalTime - (BackendTime + ClientTime)
| project TimeGenerated, TotalTime, BackendTime, ClientTime, ApimTime, CacheTime, BackendMethod, BackendUrl, BackendResponseCode, Url, ResponseCode,
LastErrorMessage, LastErrorReason, LastErrorSource, LastErrorSection, ApiId, ProductId, OperationId

(GatewayLogs schema) TotalTime - Number of milliseconds spent on overall HTTP request (from first byte received by API Management to last byte a client received back)

BackendTime - Number of milliseconds spent on overall backend I/O (connecting, sending, and receiving bytes)

ClientTime - Number of milliseconds spent on overall client I/O (connecting, sending, and receiving bytes) Similarly, you can navigate to Application Insights (configured for APIM) -> Transaction search for end-to-end transaction and review Duration.

From your description, it looks like the backend time is slow i.e. consuming the time not APIM or client but when called directly, it returns quickly. This might be due to network latency and make sure APIM instance region is closer to on-premises APIs. Do you have any network configuration/set up between APIM and on-premises?