RedisTimeoutException under high load: Timeout awaiting response

Question

I'm experiencing a RedisTimeoutException when running commands to my Azure Cache for Redis P1 instance, but only when my .NET Web API server is under high load. Here are the details of the exception:

Timeout awaiting response (outbound=1647KiB, inbound=66KiB, 7312ms elapsed, timeout is 5000ms), command=GET, next: GET xxxx, inst: 0, qu: 122, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 131072, last-in: 1079, cur-in: 1032, sync-ops: 0, async-ops: 35372, serverEndpoint: my-endpoint.redis.cache.windows.net:6380, conn-sec: 2053.26, aoc: 0, mc: 1/1/0, mgr: 9 of 10 available, clientName: my-client(SE.Redis-v2.6.111.64013), IOCP: (Busy=2,Free=998,Min=200,Max=1000), WORKER: (Busy=166,Free=32601,Min=200,Max=32767), POOL: (Threads=171,QueuedItems=1666,CompletedItems=83610), v: 2.6.111.64013 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

I've checked the article on common client-side issues that can cause timeouts (https://stackexchange.github.io/StackExchange.Redis/Timeouts), but I'm still unable to resolve the issue. Can someone please help me identify the root cause of this exception and suggest possible solutions?

Answer 1

Hi dev-thakku,

Welcome to Microsoft Q&A forum.

As I understand, you are getting timeout issue in Azure Cache for Redis when your web api server is at high load.

Timeouts happen in the client that you use to talk to Redis. When a command is sent to the Redis server, the command is queued up. The Redis server eventually picks up the command and executes it. However, the client can time out during this process. If it does, an exception is raised on the calling side. For more information on troubleshooting timeout issues, see client-side troubleshooting and StackExchange.Redis timeout exceptions.

When client connections reach the maximum for the cache, you can have failures in client requests for connections beyond the maximum. High client connections can also cause high server load when processing repeated reconnection attempts.

High client connections might indicate a connection leak in client code. Connections might not be getting reused or closed properly. Review client code for connection use.

If the high connections are all legitimate and required client connections, upgrading your cache to a size with a higher connection limit might be required. Check if the Max aggregate for Connected Clients metric is close or higher than the maximum number of allowed connections for a particular cache size. For more information on sizing per client connections, see Azure Cache for Redis performance.

Please refer to the below links for troubleshooting of timeout issue:

https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-troubleshoot-timeouts#stackexchangeredis-timeout-exceptions

and

https://azure.microsoft.com/en-us/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/

Let us know if the above links suggestions helped.

Thanks