Frequent redis timeout performing EVAL errors with asp.net redis session state provider

Larry Brandt 1 Reputation point
2020-09-25T19:58:56.86+00:00

Hello. We are using azure app services and when moving to Azure moved from inproc session state to using the asp.net Redis session state provider. It was working fine, but traffic on the site is going up and now getting more frequent errors like the one below. Have had a couple hundred of these errors in the last 24 hours. The article linked in the error doesnt help much as we are using this for session state only.

I have increased the Redis SKU a couple of times. From a basic, to C1 then C2. There are no memory/cpu pressures even on basic, but did this to rule out lack of network bandwidth on the cache. But still only looking at around 100kb/sec network on redis so i see no value in increasing the SKU anymore.
It is really interesting that memory usage has been 100MB for the last 30 days. That amount has not deviated at all, so i dont know if there is something that caps memory usage of redis?

I have also tried increasing the timeout values to larger amounts. the current settings are listed below.
App services and redis cache are all in the Central US region.

ERROR
Timeout performing EVAL (1000ms), inst: 56, qs: 21, in: 0, serverEndpoint: Unspecified/REDACTED.redis.cache.windows.net:6380, mgr: 10 of 10 available, clientName: RD0003FF6492FE, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=12,Free=32755,Min=4,Max=32767), v: 2.0.519.65453 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

WEB.CONFIG setup
<sessionState mode="Custom" customProvider="MySessionStateStore">
<providers>
<add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="REDACTED.redis.cache.windows.net" port="6380" databaseId="0" accessKey="REDACTED" throwOnError="true" applicationName="REDACTED" connectionTimeoutInMilliseconds="15000" operationTimeoutInMilliseconds="1000" retrytimeoutInMilliseconds="5000" ssl="true" />
</providers>
</sessionState>
</system.web>
<system.webServer>
<modules>
<remove name="Session" />
<add name="Session" type="Microsoft.AspNet.SessionState.SessionStateModuleAsync, Microsoft.AspNet.SessionState.SessionStateModule, Version=1.1.0.0, Culture=neutral, PublicKeyToken=REDACTED" preCondition="integratedMode" />
</modules>

Azure Cache for Redis
Azure Cache for Redis
An Azure service that provides access to a secure, dedicated Redis cache, managed by Microsoft.
213 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,869 questions
0 comments

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mike Ubezzi 2,776 Reputation points
    2020-09-28T17:26:07.677+00:00

    Hi, anonymous user - The following two documents will help you drill-down to the root cause. Please see: Troubleshoot Azure Cache for Redis server-side issues and Troubleshoot Azure Cache for Redis timeouts.

    Timeout performing EVAL (1000ms), inst: 56, qs: 21, in: 0, serverEndpoint: Unspecified/REDACTED.redis.cache.windows.net:6380, mgr: 10 of 10 available, clientName: RD0003FF6492FE, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=12,Free=32755,Min=4,Max=32767), v: 2.0.519.65453

    This link to interpret the numbers.

  2. Larry Brandt 1 Reputation point
    2020-09-28T19:28:59.057+00:00

    Those articles didnt help. but did figure it out.

    The .NET thread pool worker thread busy count = 12 while the minimum thread count = 4. Anytime busy counter is greater than the Min count for IOCP or worker threads, the .NET thread pool will throttle the creation of new threads. This can lead to Redis timeouts.

    In the Application_start method for the site set the following for minimum threads. the errors stopped immediately.

    o System.Threading.ThreadPool.SetMinThreads(200, 200);

  3. Jan Sondergaard 11 Reputation points
    2021-01-07T05:44:02.81+00:00

    anonymous user , thanks for posting this. We have been on Redis for years and also see timeout exceptions. 68 in the last 24 hours alone. It's frankly still much of a black box to us why these exceptions occur. In our case, we use ThreadPool.SetMinThreads(100, 20). Still....

    Timeout performing EVAL, inst: 1, mgr: Inactive, err: never, queue: 41, qu: 0, qs: 41, qc: 0, wr: 0, wq: 0, in: 5432, ar: 0, clientName: RD00155D70E9AD, serverEndpoint: Unspecified/REDACTED.cache.windows.net:6380, keyHashSlot: 1213, IOCP: (Busy=2,Free=998,Min=20,Max=1000), WORKER: (Busy=33,Free=32734,Min=100,Max=32767)

    We are also seeing 100MB memory usage. After rebooting, Redis will quickly zoom backup to 100MB regardless of the load.

    0 comments
  4. Vithal Bhat 1 Reputation point
    2022-08-09T05:49:54.793+00:00

    We are facing the exact same issue but our threads are well in control.
    Any other solution please.
    Also we have 3 huge key value pairs of which 1 is more than 1 mb (key is small but value is 1mb),can it be causing issues