Stackexchange Redis exception: an unknown error occurred when writing the message

Question

net core 2.1 web-api uses below redis nuget package "StackExchange.Redis" Version="2.1.55" and connects to Azure Redis P1 tier, Redis version 4.0.14

On appsettings.config,

"RedisConfiguration": {
"ConnectionString": "mycache.redis.cache.windows.net:6380,password=$$$$$$$$$F543shkerXXXXXg=,ssl=True,abortConnect=False",
"DatabaseNumber": 1
}

Receiving below exception very frequently especially when maintaining 50requests/sec. Our application deployed on Azure service fabric cluster with 3 nodes. No processor/memory pressure observed on any of the server nodes.

Message":"Redis exception while setting the value to cache: An unknown error occurred when writing the message","Level":"Error","ExceptionType":"StackExchange.Redis.RedisConnectionException

"Message":"Redis exception while getting the value from cache: An unknown error occurred when writing the message","Level":"Error","ExceptionType":"StackExchange.Redis.RedisConnectionException"

Complete stack trace for your reference:

"StackTrace":" at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2800\r\n at StackExchange.Redis.RedisBase.ExecuteSyncT in //src/StackExchange.Redis/RedisBase.cs:line 54\r\n at MyApplication.RedisCacheAdapter.<>c__DisplayClass21_1.<Put>b__0() in M:\work2\7ae5cc7915b5f803\MyApplication\RedisCacheAdapter.cs:line 199\r\n at Polly.Policy.<>c__DisplayClass144_0.<Execute>b__0(Context ctx, CancellationToken ct)\r\n at Polly.RetrySyntax.<>c__DisplayClass5_1.<Retry>b__1(Context ctx, CancellationToken ct)\r\n at Polly.Retry.RetryEngine.ImplementationTResult\r\n at Polly.RetrySyntax.<>c__DisplayClass5_0.<Retry>b__0(Action2 action, Context context, CancellationToken cancellationToken)\r\n at Polly.Policy.ExecuteInternal(Action2 action, Context context, CancellationToken cancellationToken)\r\n at Polly.Policy.Execute(Action2 action, Context context, CancellationToken cancellationToken)\r\n at MyApplication.RedisCacheAdapter.Put(String key, Byte[] bytes, TimeSpan expirytime) in M:\work2\7ae5cc7915b5f803\MyApplication\RedisCacheAdapter.cs:line 196","Exception":"An unknown error occurred when writing the message","LoggerName":"MyApplication.RedisCacheAdapter","Message":"Redis exception while setting the value to cache: An unknown error occurred when writing the message","Level":"Error","ExceptionType":"StackExchange.Redis.RedisConnectionException"

What would be reason for such error & how to fix this "StackExchange.Redis.RedisConnectionException"? Any change in redis connection string properties or redis version update would be useful?

Sometimes, below exception timeout exception as well when maintaining 50requests/sec on 3 nodes cluster

"Message":"Redis Timeout exception occurred.

Answer 1

@Vinay Gunjalli

We checked several cases for this issue in our history. There could be 2 problems:

1) Your Az Cache for Redis is not updated to latest version. Please try to update to latest version.

2) Also these types of connection resets are expected during patching, failover, VIP swapping, and scaling. They cannot be avoided, so I would recommend that your application be written such that it is resilient to such transient failures. Retry logic with some type of retry back-off may make sense, depending on your scenario. Here is the document explaining the same.

Please check the above and get back to us for any questions.

Thanks
Navtej S