Microsoft.Azure.ServiceBus.ServiceBusTimeoutException issues

Question

Hi Q&A team.
I have problem with Azure Service Bus integration and hope you can help me.
I send messages to ASB and regularly receive timeout exceptions with strange timeout value 00:00:00. I use exponetial retry on client and connection with retry starting from 1 second. I'm not sure where does ASB take this value and I can't understand if it is infrastructure regular problem or I should implement additional retry on my code side. I can't find any tips in documentation about this retry behaviour.
I use Microsoft.Azure.ServiceBus 5.1.2 package

So the question is should I fix my timeout and retry usage or everything is just an infrastructure issue?

Exception stacktrace:

Microsoft.Azure.ServiceBus.ServiceBusTimeoutException: The operation did not complete within the allocated time 00:00:00 for object sender14579. ---> System.AggregateException: One or more errors occurred. ---> System.TimeoutException: The operation did not complete within the allocated time **00:00:00** for object sender14579.  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)  
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)  
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)  
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)  
--- End of stack trace from previous location where exception was thrown ---  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
   at Microsoft.Azure.ServiceBus.Amqp.AmqpLinkCreator.<CreateAndOpenAmqpLinkAsync>d__11.MoveNext()  
   --- End of inner exception stack trace ---  
   --- End of inner exception stack trace ---  
   at Microsoft.Azure.ServiceBus.Core.MessageSender.<OnSendAsync>d__58.MoveNext()  
--- End of stack trace from previous location where exception was thrown ---  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
   at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()  
--- End of stack trace from previous location where exception was thrown ---  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()  
--- End of stack trace from previous location where exception was thrown ---  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
   at Microsoft.Azure.ServiceBus.Core.MessageSender.<SendAsync>d__45.MoveNext()  
--- End of stack trace from previous location where exception was thrown ---  
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
   at CCWolverine.Messaging.ServiceBusAccessor.SendMessage[T](MessageToSend`1 messageToSend)  
   at CCWolverine.Model.Product.ProductNameUpdater.SendProductChangedMessages(IEntityWriter writer) in D:\a\1\s\CCWolverine.Model\Model\Product\ProductNameUpdater.cs:line 400  

Code used to sent message:

 // https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus  
        private readonly RetryPolicy _exponentialRetry = new RetryExponential(  
            TimeSpan.FromSeconds(1),  
            TimeSpan.FromSeconds(30),  
            TimeSpan.FromSeconds(1.5),  
            3);  
  
        /// <inheritdoc />  
        public MessageSender GetMessageSender(string topicName)  
        {  
            if (string.IsNullOrWhiteSpace(topicName))  
                throw new ArgumentException("Value cannot be null or whitespace.", nameof(topicName));  
  
            EnsureConnectionIsAlive();  
  
            var messageSender = new MessageSender(_connection, topicName, _exponentialRetry);  
  
            return messageSender;  
        }  
  
        /// <summary>  
        ///     We reinitialize connection in case of connection failure  
        ///     Use of Double-Check-Locking to optimize  
        /// </summary>  
        private void EnsureConnectionIsAlive()  
        {  
            if (_connection == null || _connection.IsClosedOrClosing)  
            {  
                lock (Locker)  
                {  
                    if (_connection == null || _connection.IsClosedOrClosing)  
                    {  
                        var connection = new ServiceBusConnection(_serviceBus.Settings.Url)  
                        {  
                            RetryPolicy = _exponentialRetry  
                        };  
  
                        _connection = connection;  
                    }  
                }  
            }  
        }  
  
        /// <inheritdoc />  
        public bool SendMessage<T>(MessageToSend<T> messageToSend) where T : WolverineMessage  
        {  
            if (messageToSend == null)  
                throw new ArgumentNullException(nameof(messageToSend));  
  
            var topicClient = ClientFactory.GetMessageSender(messageToSend.To);  
  
            var message = messageToSend.Data.ToMessage().WithLabel(_label);  
  
            topicClient.SendAsync(message)  
                .ConfigureAwait(false)  
                .GetAwaiter()  
                .GetResult();  
  
            topicClient.CloseAsync()  
                .ConfigureAwait(false)  
                .GetAwaiter()  
                .GetResult();  
  
            return true;  
        }  

Answer 1

Hi @JayaC-MSFT ,
No, it's not the first time I'm experiencing this issue, it starts from the beginning of our ASB usage (maybe two or three weeks). It usually happens periodically, for example everything is fine all day but from 7 to 9 am we have dozens of timeout exceptions but I think it's related to MQ load.
I didn't set any timeout value because in Microsoft.Azure.ServiceBus 5.1.2 package timeouts on MessageSender and ServiceBusConnection marked as obsolete and it's recommended to use RetryPolicy.
I used the same RetryPolicy for connection and for sender from https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus.

private readonly RetryPolicy _exponentialRetry = new RetryExponential(             TimeSpan.FromSeconds(1),             TimeSpan.FromSeconds(30),             TimeSpan.FromSeconds(1.5),             3);	  

I don't think that there are any firewalls, because most messages are delivered without problems. I've sent you more information into a private channel.

Answer 2

@F V Same problem here, have you solved it yet ?

Answer 3

same problem here! it happens once in a blue moon, even if we do not have any messages send or received from the date we started the app till this happen.