Why Is My ServiceBusClient Token Expiring?

Question

I have an application written in C# that processes messages placed on an Azure Service Bus Queue. This application is a BackgroundService that is started using the AddHostedService method.

I am creating a credentials object using a ClientSecretCredential object:

credentials = new ClientSecretCredential(
            azureCreds.TenantId,
            azureCreds.ClientId,
            azureCreds.ClientSecret);

This is then passed to the ServiceBusQueueClient constructor, and then a processor is created.

client = new ServiceBusClient(azureNamespace, credentials, clientOptions);
processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());

I add handlers for the messages:

processor.ProcessMessageAsync += Processor_ProcessMessageAsync;
processor.ProcessErrorAsync += Processor_ProcessErrorAsync;

Everything works well for a couple of hours. Messages are received and processed. Then I start getting the following token timeout error:

System.UnauthorizedAccessException: Put token failed. status-code: 401, status-description: ExpiredToken: The token is expired. TrackingId:<redacted>, SystemTracker:NoSystemTracker, Timestamp:<redacted>. For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot.    at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.CreateReceivingLinkAsync(String entityPath, String identifier, AmqpConnection connection, Uri endpoint, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenReceiverLinkAsync(String identifier, String entityPath, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenReceiverLinkAsync(TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String identifier, CancellationToken cancellationToken)    at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)    at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)    at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.ReceiveMessagesAsyncInternal(Int32 maxMessages, Nullable`1 maxWaitTime, TimeSpan timeout, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.ReceiveMessagesAsyncInternal(Int32 maxMessages, Nullable`1 maxWaitTime, TimeSpan timeout, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c.<<ReceiveMessagesAsync>b__45_0>d.MoveNext() --- End of stack trace from previous location ---    at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken, Boolean logTimeoutRetriesAsVerbose)    at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken, Boolean logTimeoutRetriesAsVerbose)    at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.ReceiveMessagesAsync(Int32 maxMessages, Nullable`1 maxWaitTime, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.ServiceBusReceiver.ReceiveMessagesAsync(Int32 maxMessages, Nullable`1 maxWaitTime, Boolean isProcessor, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.ReceiverManager.ReceiveAndProcessMessagesAsync(CancellationToken cancellationToken) 

I have searched online, but everything appears to indicate that I do not have to refresh the token manually, that the Microsoft Identity library will handle that.

Has anyone else experienced this issue? Do I have to refresh the token periodically? If so, how do I know it is expiring since the expiration information is not accessible in the Service Bus Client object?

Answer 1

@Bob McGowan Thanks for posting your question in the Microsoft Q&A forum.

I'm glad that you were able to resolve your issue and thank you for posting your solution in your SO thread so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue: you were getting below error System.UnauthorizedAccessException: Put token failed. status-code: 401, status-description: ExpiredToken: The token is expired.

Solution: Since your application was running as a Windows service it was set to "Automatic" startup. But the server it was running on would boot in UTC time, then the Windows time service would change it to EST a minute or so later. Unfortunately, your service had already started so the token expiration would get confused. By changing Windows service to start as "Automatic (Delayed Start)" it gave the time service enough time to update the system time, and everything worked properly afterwards.

Please feel free to comment below with any additional information.