Failed to initialize Integration Runtime (Self-hosted) node components

Question

We recently configured our datafactory to collect data from 60 different branches of our business. We created 60 different integration runtime via the Data Factory UI and then installed IR on a local computer in each branch and copied over the registration key.

59 of these worked perfectly. On one single case though I get an error that says

"The integration Runtime (Self-hosted) node has encountered an error during registration.
Failed to initialize Integration Runtime (Self-hosted) node components. You may check event log for more detail."

I'm trying to debug this issue quickly if possible. I'm hoping it's something simple like a firewall configuration, I could not find this specific error anywhere on the internet. Has anyone seen this before?

The registration key is recognized on the runtime in question but then when I click "Finish" it just stays stuck in "initializing...." for a long time and then throws the error.

I tried uploading the xml log but for some reason I am unable to upload a small 80kb file. I went through the log myself and found that there's a few errors that occur when I face the above issue:

Authentication Key is not assigned yet.  
---------------  
Pre-validate token failed. Error code: AuthSasEffectiveInFuture   
---------------  
DEBUG: Pre-validate token failed. Error code: AuthSasEffectiveInFuture Activity ID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : Exception detail: Microsoft.DataTransfer.AuthService.Models.KmsApiException: ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : at Microsoft.DataTransfer.AuthService.Models.ClientTokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.HighAvailability.TokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.PreValidateToken(ConfigVariableManager configManager)  
---------------  
DEBUG: Failed in Retry Stage. ActivityID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d Activity ID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : Exception detail: Microsoft.DataTransfer.AuthService.Models.KmsApiException: ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : at Microsoft.DataTransfer.AuthService.Models.ClientTokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.HighAvailability.TokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.PreValidateToken(ConfigVariableManager configManager) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.Run(ConfigVariableManager configManager, CancellationToken ct) at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext()  
  
-----------  
  
Create cloud endpoint listener failed. TransferMode: Streamed,   
  
-----------  
  
DEBUG: Create cloud endpoint listener failed. TransferMode: Streamed, Endpoint Information: e35c1065-eac7-4295-9cd4-a0a8b86919d6. Activity ID: 56b00c61-3aad-4178-88b4-42bfc37dcd98 Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' Exception detail: Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)  
  
  
  
-------------------  
  
DEBUG: Unable to create cloud enpdoint listener. Error message: One or more errors occurred.. Activity ID: 56b00c61-3aad-4178-88b4-42bfc37dcd98 Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' Exception detail: System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList) --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.SpecifyServiceBusAddressList(ReadOnlyCollection`1 sbDataList) at Microsoft.DataProxy.MidgardCore.ConnectToServiceBusStage.<Run>d__4.MoveNext() ---> (Inner Exception #0) Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)<--- ---> (Inner Exception #1) Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)<---   
  

Answer 1

Hi again!

I fixed the issue, I tried making sense of those logs and "AuthSasEffectiveInFuture" this caught my eye. I followed a hunch and checked timezone settings on that computer and realized the branch had been set to an incorrect timezone (they are UTC -0700 but they were set to UTC -0500) so to correct that the branch manager and manually edited the time to 2 hours back.

This causes the authentication between Azure and the node to break.

I feel so bad to have wasted my entire day and your time on such a silly issue, had the error code from Azure been "Check your system time" life would have been a little easier.

Thanks so much for trying to help!

Answer 2

Hi @Sunil_CATO ,

Welcome to Microsoft Q&A and thanks for reaching out.

Are you seeing this error consistently or a transient one? I would also recommend you to please check event logs for more information about the root cause of the issue. If possible please do attach the log file.

Also, please make sure Remote access to intranet option is enabled on the node

I see that you have created 60 SHIR in your ADF. Ss there any specific reason you opted out to created multiple nodes for each Integration Runtime?
You can associate a self-hosted integration runtime with multiple on-premises machines or virtual machines in Azure. These machines are called nodes. You can have up to four nodes associated with a Single self-hosted integration runtime. Using node you can reduce your Integration Runtime count from 60 IR's/4 nodes = 15 IR's which will also help you maintain a High availability and scalability (to achieve higher performance). For more info please refer to :

• High availability and scalability
• Self-hosted integration runtime scalability

Hope this helps.

----------

Thank you

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members.