Azure API Management latency of over 5 seconds

Sateesh Anagani 6

We are seeing APIM latency spikes morethan 5s, when we check details roundtrip latency utilizing more time.
observed during the spike AKS pods were restarting and Disk IO for particular node we can see spike at the same time.
when we check pod logs its showing due to probe failure.

please help on that what is causing issue. we were not able to find resolution.

Luke Murray 10,611 Reputation points MVP

2022-09-29T03:10:06.713+00:00

Log an Azure support case, https://azure.microsoft.com/en-us/support/create-ticket/

The Customer Support team, can look at the backend logs - to help find the source of the latency.
Luke Murray 10,611 Reputation points MVP

2022-09-29T03:13:19.87+00:00

Also (its a bit outdated), but take a look at this APIM policy: https://learn.microsoft.com/en-us/archive/blogs/cie/detect-backend-api-response-latency-using-apim-policy - to help with comparisons.

1 answer

MuthuKumaranMurugaachari-MSFT 22,236 Reputation points

2022-09-30T14:56:55.037+00:00

@Anonymous Thank you for reaching out to Microsoft Q&A. Based on your statement, I assume you have AKS Pods as a backend service for APIM and found Pods were having some issues around that time (during spike). Is my understanding correct?

Do you have diagnostics settings configured for the APIM instances? If so, you can review Resource logs to understand where the latency occurs. ApiManagementGatewayLogs provides the following info and from which you can calculate APIM time = TotalTime - BackendTime - ClientTime.

BackendTime - Number of milliseconds spent on overall backend I/O (connecting, sending, and receiving bytes)
ClientTime - Number of milliseconds spent on overall client I/O (connecting, sending, and receiving bytes)
TotalTime - Number of milliseconds spent on overall HTTP request (from first byte received by API Management to last byte a client received back)

Based on that, you can troubleshoot if the problem occurs in AKS Pods or APIM. Please review and share the details. If you haven't enabled diagnostics settings, azure support team can help you investigate this from our backend logs. I would recommend creating one or let me know if you don't have a support plan. I can assist you with one-time free technical support.
Please sign in to rate this answer.
Sateesh Anagani 6 Reputation points

2022-09-30T19:01:10.993+00:00

MuthuKumaranMurugaachari-MSFT
thanks for your Answer,
Yes you are correct backend service(AKS) found issues during spike.and also we observed backend time and round trip latency time are high from the logs.
please let me know if any queries to check backend time and client time its would be great for me.

MuthuKumaranMurugaachari-MSFT 22,236 Reputation points

2022-09-30T19:19:49.977+00:00

@Anonymous I am glad that it helped you. For querying diagnostic logs (to enable Resource Logs), you can navigate to APIM instance in azure portal -> Logs and there are many sample queries to pick from. You can try sample like below for checking backend time and client time:

ApiManagementGatewayLogs
| where TimeGenerated > ago(1d)
| extend ApimTime = TotalTime - (BackendTime + ClientTime)
| project TimeGenerated, TotalTime, BackendTime, ClientTime, ApimTime, CacheTime, BackendMethod, BackendUrl, BackendResponseCode, Url, ResponseCode,
LastErrorMessage, LastErrorReason, LastErrorSource, LastErrorSection, ApiId, ProductId, OperationId
| render timechart

You can look for KQL Queries and Sample Queries docs which would help in writing more filter criteria. I hope this answers your question and feel free to add if you have any questions. I would be happy to assist you.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community.

Amit-J 316 Reputation points

2022-10-31T09:52:27.553+00:00

Hi @MuthuKumaranMurugaachari-MSFT

MS Docs suggest other metrics for time like duration, backed time etc n the link - https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-use-azure-monitor#view-diagnostic-data-in-azure-monitor

Here, can you please help me understand how to derive gateway time , because even in my apiminstance, I always get duration = backend time + client time.. where is gateway processing time ?

Amit-J 316 Reputation points

2022-10-31T12:51:17.85+00:00

and a lot of times durationMs is less than backendtime ....
and sometimes both are same with "clienttime" missing from the logs.
and sometimes durationMs = backendtime + clienttime

Please let me know if I need to start another thread for this.

MuthuKumaranMurugaachari-MSFT 22,236 Reputation points

2022-10-31T13:48:28.12+00:00

@Amit-J To calculate APIM time (or Gateway), use ApimTime = TotalTime - (BackendTime + ClientTime) in your query. Please start a new thread with your questions and would like to answer them.
Sign in to comment

Share via

Azure API Management latency of over 5 seconds

1 answer