This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 89%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Hi,
We have recently implemented Azure Redis Cache with .Net Framework 4.7.2. It was working fine. Suddenly we are keep getting the timeout exceptions in StackExchange.Redis for Azure Redis Cache.
Exception:
Timeout performing EVAL (5000ms), next: EVAL, inst: 22, qu: 0, qs: 18, aw: False, rs: ReadAsync, ws: Idle, in: 131072, serverEndpoint: Unspecified/KreatoCRMCache.redis.cache.windows.net:6380, mgr: 10 of 10 available, clientName: NextGenITServices_IN_0, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=8171,Min=4,Max=8191), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Stack Trace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2250 at StackExchange.Redis.RedisBase.ExecuteSyncT in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 1134 at Microsoft.Web.Redis.StackExchangeClientConnection.<>c__DisplayClass7_0.b__0() in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 68 at Microsoft.Web.Redis.StackExchangeClientConnection.OperationExecutor(Func1 redisOperation) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 95 at Microsoft.Web.Redis.StackExchangeClientConnection.RetryLogic(Func1 redisOperation) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 118 at Microsoft.Web.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 68 at Microsoft.Web.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(DateTime lockTime, Int32 lockTimeout, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisConnectionWrapper.cs:line 184 at Microsoft.Web.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContextBase context, String id, CancellationToken cancellationToken, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 291 at Microsoft.Web.Redis.RedisSessionStateProvider.d__20.MoveNext() in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 206 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.d__74.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.d__65.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.TaskAsyncHelper.EndTask(IAsyncResult ar) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.EndAcquireState(IAsyncResult result) at System.Web.HttpApplication.AsyncEventExecutionStep.InvokeEndHandler(IAsyncResult ar) at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar)
Plan: Basic 1 GB (C1)
Web Configuration:
Hi @Venkatesh K which region is your Redis deployed to and when did your timeouts start to occur? We can also work more closely with you by sending us the name of your Redis instance to AzCommunity[at]microsoft[dot]com ATTN Ryan.
Hi Ryan, I've sent a mail about Redis cache details to AzCommunity[at]microsoft[dot]com.
Redis Intance Name: KreatoCRMCache.redis.cache.windows.net
Location: Southeast Asia (Both Redis and Web Instance)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 10%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EX%3C/text%3E%3C/svg%3E)
I did some research and it looks to be pretty nasty, I leave you with a link
which came from this link
https://github.com/StackExchange/StackExchange.Redis/issues/508
Hi, I've investigated the details using above link which you shared. We have implemented with correct manner only.
Cache Name: KreatoCRMCache.redis.cache.windows.net:6380
Cache Size: Basic 1 GB(c1)
Public Virtual IP (VIP) Address of client deployments: 23.97.51.212
Version of StackExchange.Redis: 2.0.601, Microsoft.Web.RedisSessionStateProvider: 4.0.1
Cache service and clients Region: Southeast Asia
Number of client instances: only 1 ( our clients are very less. max 100 clients)
CPU utilization on our client both before and during the incident: Its max: 25-30% only
Client for sync timeout and for connection timeout: connectionTimeoutInMilliseconds="5000", operationTimeoutInMilliseconds="1000" & retryTimeoutInMilliseconds="3000"
@Venkatesh K , take a look at https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md which actually refers to the link @xequence posted. The timeout you're encountering might be related to threadpool throttling. file-threadpool.md has a gist that recommends some best practices so that you can do to avoid hitting throttling issues.
Let me know if this helps.
Shall we increase the "minIoThreads" 4 to 50 & "minWorkerThreads" 4 to 50? Ours is 4 core machine.
That seems like a good start @Venkatesh K . I would make sure to do the changes incrementally and check for improvements.
@Venkatesh K , wanted to follow up to see if you were successful in increasing thread count settings.
Hi @Ryan Hill , we are making the process by next week only. Now we are in vocation. Lets implement this and will check for times then will let you know by this month end(Jan20). Don't close the ticket, please keeps its open till my updates.