Error on Self Hosting Integrated Runtime

Question

Error on Self Hosting Integrated Runtime

Jed Fletcher 6

I have two nodes running on my integrated runtime however during an update the the self hosted IR's one of the nodes after updating threw an error of "Authentication Key Invalid" so we uninstalled the runtime on that server and re-installed and registered it using the key. This is all we keep getting:

This was working fine before the update what would be causing this issue. I have looked though the logs and found the following:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  
- <System>  
  <Provider Name="Integration Runtime (Self-hosted) Host Service" />   
  <EventID Qualifiers="0">54</EventID>   
  <Level>2</Level>   
  <Task>0</Task>   
  <Keywords>0x80000000000000</Keywords>   
  <TimeCreated SystemTime="2021-05-20T19:25:05.892234400Z" />   
  <EventRecordID>129186086</EventRecordID>   
  <Channel>Integration Runtime</Channel>   
  <Computer>xxxxx</Computer>   
  <Security />   
  </System>  
- <EventData>  
  <Data>DEBUG: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.DataProxy.MidgardCore.StartupStage.StageRetryException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException: Exception of type 'Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException' was thrown. at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcess.Start() at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<>c__DisplayClass40_2.<<StartLocalWorkerProcessesAsync>b__2>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartLocalWorkerProcessesAsync>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartAsync>d__31.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.InitWorkerProcessManager(WorkerProcessSettings settings, ConfigVariableManager configManager, String communicationUri, Int32 workerCount, Boolean isGrpcEnabled) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.Run(ConfigVariableManager configManager, CancellationToken ct) at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext() --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext() --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.StartupStage.CoreInitStageParallelSync.<Run>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.DataProxy.MidgardCore.StartupStage.StageMachineSequenceSync.<Run>d__0.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.DataProxy.MidgardCore.ServiceHostWrapperAsync.Open() at Microsoft.DataProxy.MidgardCore.HybridProxyAgent.OnStart(IHostServiceProvider provider, StartStatus startStatus) ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Microsoft.DataProxy.MidgardCore.StartupStage.StageRetryException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException: Exception of type 'Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException' was thrown. at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcess.Start() at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<>c__DisplayClass40_2.<<StartLocalWorkerProcessesAsync>b__2>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartLocalWorkerProcessesAsync>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartAsync>d__31.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.InitWorkerProcessManager(WorkerProcessSettings settings, ConfigVariableManager configManager, String communicationUri, Int32 workerCount, Boolean isGrpcEnabled) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.Run(ConfigVariableManager configManager, CancellationToken ct) at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext() --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext() --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.StartupStage.CoreInitStageParallelSync.<Run>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.DataProxy.MidgardCore.StartupStage.StageMachineSequenceSync.<Run>d__0.MoveNext() ---> (Inner Exception #0) Microsoft.DataProxy.MidgardCore.StartupStage.StageRetryException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException: Exception of type 'Microsoft.DataTransfer.WorkerProcess.Common.WorkerProcessStartupException' was thrown. at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcess.Start() at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<>c__DisplayClass40_2.<<StartLocalWorkerProcessesAsync>b__2>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartLocalWorkerProcessesAsync>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.DataTransfer.Worker.ProcessManager.LocalWorkerProcessManager.<StartAsync>d__31.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.InitWorkerProcessManager(WorkerProcessSettings settings, ConfigVariableManager configManager, String communicationUri, Int32 workerCount, Boolean isGrpcEnabled) at Microsoft.DataProxy.MidgardCore.StartupStage.StartLocalWorkerProcessManagerStage.Run(ConfigVariableManager configManager, CancellationToken ct) at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext() --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext()<--- <---</Data>   
  </EventData>  
  </Event>

1 answer

Your answer

Answer 1

KranthiPakala-MSFT 46,632 Microsoft Employee

Hi @Jed Fletcher ,

Thanks for reaching out and using Microsoft Q&A forum.

When you reinstalled the IR in the machine that it is failing, have you reinstalled the same version that is there in the other node? If not, please make sure both the SHIR version are matching on the VMs. Also please make sure that the credentials between the both the nodes are in sync. if you think they are not in sync, then you can generate credential back from the working node and import into the node that is having issue.

Do let us know how it goes.

Thanks

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Jed Fletcher 6 Reputation points

2021-05-24T14:24:06.423+00:00

The issue is that the second node cant connect to the first node. We looked at the settings on the first node and the change button is greyed out that allows us to enable the nodes to connect to each other.
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2021-05-25T00:32:34.043+00:00

Hi @Jed Fletcher , thanks for your response. Could you please confirm if the issue still persists or resolved?

Thanks
Jed Fletcher 6 Reputation points

2021-05-25T12:37:09.88+00:00

The issue has not been resolved like I showed you in the image we are unable to enable the remote access those radio boxes are grayed out.

Is there a way to enable those?
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2021-05-26T06:45:04.87+00:00

Thanks for your response and confirmation @Jed Fletcher . As a work around have you got a chance to try uninstalling and re-installing/reconfiguring the SHIR on first node? If not, would you please give a try and see if that helps to resolve the issue? Sorry, unfortunately I wasn't able to reproduce the issue to check for alternatives hence suggesting to reinstall/reregister the SHIR on first node.

Do let us know how it goes.

Thank you.
Jed Fletcher 6 Reputation points

2021-05-26T14:16:18.787+00:00

I am unable to get authorization to re-install the first node is there a config file I can edit to increase the memory recognition.
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2021-05-28T07:25:20.463+00:00

Hi @Jed Fletcher , Not that I am aware of but as per latest info from internal team, changing remote access settings is only allowed when there is one node in cluster. In old SHIR version, if registering node failed, there might be some orphan node leaking, resulting in showing two nodes in IR cluster on portal. Could you please check how many nodes in this IR on portal and deleted these orphan data. In latest SHIR version > 5.4.7749.1, no orphan node is leaked if registering failed.

In case you are still blocked, for deeper investigation and immediate assistance, If you have a support plan could you file a support ticket? If you don't have a support plan do let us know so that we can work with you on opening a support ticket.

Do let us know how it goes.
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2021-06-01T20:10:49.29+00:00

Hi @Jed Fletcher , As we still have not heard back from you. Just wanted to check if you are still facing the issue or need assistance on this? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Share via

Error on Self Hosting Integrated Runtime

1 answer

Your answer