SCOM Management Server Greyed Out

Question

Hi All,

We are using SCOM 2019 UR3 Version, we have 4 Management Servers, one MS got greyed out. We are observing critical state in Data Access Service Group in the operations manager group health state pane.

I tried restarting the System Center data Access service on the affected MS, below error appeared in SCOM console.

I checked the connectivity of all Management Servers over port 5724, they are all connecting. I also cleared cleared cache after on the greyed MS, still it is not getting healthy.

The services are in running state only.

Kindly check and suggest any fixes.

Thanks in advance.

Regards

Fadil CK

Note:  The following information was gathered when the operation was attempted.  The information may appear cryptic but provides context for the error.  The application will continue to run.

Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://MS2xxxx:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0035207. TCP error code 10061: No connection could be made because the target machine actively refused it xx.xx.xx.xx:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it xx.xx.xx.xx:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace: 
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.LayeredChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Reconnect(EnterpriseManagementConnectionSettings connectionSettings)
   at Microsoft.EnterpriseManagement.EnterpriseManagementGroupInternal.Reconnect()
   --- End of inner exception stack trace ---
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
   at Microsoft.EnterpriseManagement.EnterpriseManagementGroupInternal.Reconnect()
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.TaskHelper.GetFavoriteTasks(IList`1 categoryIds, ManagementGroup group)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.MomViewBase.FavoriteTasksStatusJob(Object sender, ConsoleJobEventArgs args)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(IComponent component, EventHandler`1 job, Object sender, ConsoleJobEventArgs args)
System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://ldprdscomms1dc2:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0035207. TCP error code 10061: No connection could be made because the target machine actively refused it xx.xx.xx.xx:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it xx.xx.xx.xx:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace: 
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.LayeredChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Reconnect(EnterpriseManagementConnectionSettings connectionSettings)
   at Microsoft.EnterpriseManagement.EnterpriseManagementGroupInternal.Reconnect()
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it xx.xx.xx.xx:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

Answer 1

Hi,

This issue has been fixed, this was due to some changes in group policies was done, causing the SCOM action account permission altered in the management servers.

The policies were changed and required permissions was given to the action accounts and the SCOM management servers are now up and running.

Thanks

Answer 2

Hi,

For this issue, please review the Operations Manager event log first.

Please confirm if the UDL test is successful from SCOM management server to SQL server.

How to do a UDL test:

• 1. Create a text file and save it with .udl extension
• 2. Now provide the SQL server instance name which is hosting your SCOM OperationsManager database
• 3. Now select windows authentication and test the connection.

Try to enable verbose tracing for Management Server

SCOM VERBOSE TRACING