SCOM DB issue

Question

SCOM DB issue

py 206

Hello All,

Any advise will be much appreciated.

Here is the issue. We have a 2 node cluster DB server, Always ON
When installing new MS i used Server A to detect OperationsManager DB (My bad). Now when the server has failed over to Node B the data access service is stopped and unable to start it.

I have changed the name of the Database server name to Cluster name in the registry and restarted the server without success. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup

I have checked for the password of sdk service and it is ruled out as cause of the issue. The permissions for SDK service account is identical on both the DB server nodes and listener.

Below is the error from Management Server. I'm assuming since the DB server has failed over and the node A is in read only mode it is unable to start the service.

Log Name: Operations Manager
Source: OpsMgr SDK Service
Date: 8/19/2020 7:17:32 AM
Event ID: 26380
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
The System Center Data Access service failed due to an unhandled exception.
The service will attempt to restart.
Exception:

Microsoft.EnterpriseManagement.ConfigurationReaderException: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ClientSessionManager, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container. ---> Microsoft.EnterpriseManagement.ConfigurationReaderException: The component named 'QueueRegistration' and of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container. ---> System.Data.SqlClient.SqlException: Failed to update database "OperationsManager" because the database is read-only.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.ExecuteReader(ExecuteArguments executeArguments, QueryResults& queryResults)
at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.Execute[T](ExecuteArguments executeArguments, RetryPolicy retryPolicy, GenericExecute1 genericExecute) at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.ExecuteReader(SqlCommand sqlCommand, IList1 prologEpilogList, IList1 projection, QueryDefinition queryDefinition, RetryPolicy retryPolicy) at Microsoft.EnterpriseManagement.DataAccessLayer.QueryRequest.Execute(SqlNotificationRequest sqlNotificationRequest) at Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService.SetupBrokerService(String serviceName, String queueName) at Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService.Initialize(IContainer container) at Microsoft.EnterpriseManagement.SingletonLifetimeManager1.GetComponentK
at Microsoft.EnterpriseManagement.Container.GetServiceInternal[T](Type type, String name)
at Microsoft.EnterpriseManagement.Container.AddService[T,V](String name, ActivationContext1 context) --- End of inner exception stack trace --- at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ProcessComponent(String componentId, XPathNavigator componentNavigator, IDictionary2 pendingComponents, IContainer container)
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadComponents(XPathNavigator navi, IContainer container)
at Microsoft.EnterpriseManagement.ServiceDataLayer.ClientSessionManager.Initialize(IContainer container)
at Microsoft.EnterpriseManagement.SingletonLifetimeManager1.GetComponent[K]() at Microsoft.EnterpriseManagement.FeatureContainer.GetFeatureInternal[T](Type type, String featureName) at Microsoft.EnterpriseManagement.FeatureContainer.AddFeatureInternal[T,V](ActivationContext1 context, String featureName)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadFeatures(XPathNavigator navi, IContainer container)
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.Process()
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(InProcEnterpriseManagementConnectionSettings configuration)
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="OpsMgr SDK Service" />
<EventID Qualifiers="49152">26380</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-08-19T12:17:32.197395000Z" />
<EventRecordID>7306213</EventRecordID>
<Channel>Operations Manager</Channel>
<Computer></Computer>
<Security />
</System>
<EventData>
<Data>Microsoft.EnterpriseManagement.ConfigurationReaderException: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ClientSessionManager, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container. ---> Microsoft.EnterpriseManagement.ConfigurationReaderException: The component named 'QueueRegistration' and of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container. ---> System.Data.SqlClient.SqlException: Failed to update database "OperationsManager" because the database is read-only.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.ExecuteReader(ExecuteArguments executeArguments, QueryResults& queryResults)
at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.Execute[T](ExecuteArguments executeArguments, RetryPolicy retryPolicy, GenericExecute1 genericExecute) at Microsoft.EnterpriseManagement.DataAccessLayer.SqlRetryHandler.ExecuteReader(SqlCommand sqlCommand, IList1 prologEpilogList, IList1 projection, QueryDefinition queryDefinition, RetryPolicy retryPolicy) at Microsoft.EnterpriseManagement.DataAccessLayer.QueryRequest.Execute(SqlNotificationRequest sqlNotificationRequest) at Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService.SetupBrokerService(String serviceName, String queueName) at Microsoft.EnterpriseManagement.ServiceDataLayer.QueueRegistrationService.Initialize(IContainer container) at Microsoft.EnterpriseManagement.SingletonLifetimeManager1.GetComponentK
at Microsoft.EnterpriseManagement.Container.GetServiceInternal[T](Type type, String name)
at Microsoft.EnterpriseManagement.Container.AddService[T,V](String name, ActivationContext1 context) --- End of inner exception stack trace --- at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ProcessComponent(String componentId, XPathNavigator componentNavigator, IDictionary2 pendingComponents, IContainer container)
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadComponents(XPathNavigator navi, IContainer container)
at Microsoft.EnterpriseManagement.ServiceDataLayer.ClientSessionManager.Initialize(IContainer container)
at Microsoft.EnterpriseManagement.SingletonLifetimeManager1.GetComponent[K]() at Microsoft.EnterpriseManagement.FeatureContainer.GetFeatureInternal[T](Type type, String featureName) at Microsoft.EnterpriseManagement.FeatureContainer.AddFeatureInternal[T,V](ActivationContext1 context, String featureName)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadFeatures(XPathNavigator navi, IContainer container)
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.Process()
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(InProcEnterpriseManagementConnectionSettings configuration)
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)</Data>
</EventData>
</Event>

Accepted answer

2 additional answers

Your answer

Answer 1

SChalakov 10,576 MVP Volunteer Moderator

Hi @peewhy-0274,

I think I can help here because I had a similar case with a customer of mine, who did the same. How I was able to resolve it? Perform a "Database Move" procedure by updating all the configurations. What you need to do is to update the "Node1\SCOM" instance configuration with the "Cluster(Listener)\SCOM" configuration.
The best way would be to follow those two:

How to move the Operational database
and
How to move the Reporting data warehouse database

The DB config is not stored only in the registry and in the config file on the Mangement Server, but also in different DB tables. The articles will help you identify all the places, where you can adjust (update) the DB configuration.
Using this approach we were able to solve the exact same challenge.

Hope I was able to help. Regards,
Stoyan
(If the reply was helpful please don't forget to upvote or accept as answer, thank you)

py 206 Reputation points

2020-08-20T06:12:22.66+00:00

Thanks Stoyan.

I was able to resolve once i changed the registry entry in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\System Center\2010\Common\Database to the Listener name.
SChalakov 10,576 Reputation points MVP Volunteer Moderator

2020-08-20T11:54:54.337+00:00

Hi py,
that is cool! Thanks for sharing!

Regards,
Stoyan

Answer 2

Leon Laude 86,026

Hi,

Which version of SCOM are you using? Did you go through the official documentation below?

How to configure Operations Manager to communicate with SQL Server
https://learn.microsoft.com/en-us/system-center/scom/manage-sqlserver-communication?view=sc-om-2019

PS: Please hide any sensitive information in your post.

----------

(If the reply was helpful please don't forget to upvote or accept as answer, thank you)

Best regards,
Leon

py 206 Reputation points

2020-08-20T04:34:10.953+00:00

Thanks Leon .

We are using SCOM 2016 UR4.

My bad. Sensitive matter hidden

Answer 3

Crystal-MSFT 53,991 Microsoft External Staff

Hi,

From your description, I know the System Center Data Access service is failed to start. The issue happens when the SQL DB Operations Manager failover from Server A to Node B.

In the event log, it mentioned that we are Failed to update database "OperationsManager",the database is read-only.

Based on my research, every availability replica is assigned an initial role-either the primary role or the secondary role, which is inherited by the availability databases of that replica. The role of a given replica determines whether it hosts read-write databases or read-only databases. One replica, known as the primary replica, is assigned the primary role and hosts read-write databases, which are known as primary databases. At least one other replica, known as a secondary replica, is assigned the secondary role. A secondary replica hosts read-only databases, known as secondary databases.
https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/overview-of-always-on-availability-groups-sql-server?view=sql-server-2017#AGsARsADBs

I think our issue is that after failover, the database is still in read-only mode, which cause SCOM failed to write anything into the database and then the System Center Data Access failed to start.

Given the situation, we can manually move the database to see if it is working. Or we can contact SQL support to check on SQL always on availability group failover issue.

Hope it can help.

py 206 Reputation points

2020-08-20T06:10:07.2+00:00

Thanks Crystal - Appreciate your time
Crystal-MSFT 53,991 Reputation points Microsoft External Staff

2020-08-21T06:27:23.873+00:00

Thanks for the update. I am glad to hear that the issue is resolved by changing the registry entry in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\System Center\2010\Common\Database to the Listener name. Thanks for the sharing. It can help others you faced the same issue. I appreciate it.

Again thanks for your time and have a nice day!

AdamMohamed-3032 61

@SChalakov and @Crystal-MSFT Im also facing same issue but we have update database listener name in registry. We are using SCOM 2019 and UR4.

Please note we have only one Management Server and we have upgraded our SQL server from 2016 to 2019 recently. This is our Test environment.

We have followed the instruction given in the below article :

https://learn.microsoft.com/en-us/system-center/scom/manage-move-opsdb?view=sc-om-2019

https://learn.microsoft.com/en-us/system-center/scom/manage-move-omdwdb?view=sc-om-2019

Error Event 1 :

The System Center Data Access service failed due to an unhandled exception.  
The service will attempt to restart. 
Exception: 

Microsoft.EnterpriseManagement.Common.SdkServiceNotInitializedException: The Data Access service has not yet initialized. Please try again.
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.get_Container()
   at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.SdkChannel.Start()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Error Event 2 :

The SDK service keeps stopping.

OpsMgr Management Configuration Service failed to communicate with System Center Data Access Service due to the following exception

System.Runtime.Remoting.RemotingException: Unable to get ISdkService interface. Please make sure local Sdk Service is running.
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.CredentialDataProvider.CreateSdkConnector()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.CredentialDataProvider.GetSecureData(Guid agentId, ICollection`1 addedReferenceList, ICollection`1 deletedReferenceList, ICollection`1 addedCredentialList, ICollection`1 deletedCredentialList, Byte[]& hashValue)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingCredentialDataProvider.GetSecureData(Guid agentId, ICollection`1 addedReferenceList, ICollection`1 deletedReferenceList, ICollection`1 addedCredentialList, ICollection`1 deletedCredentialList, Byte[]& hashValue)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentConfigurationFormatter.WriteSecureData(AgentConfigurationStream stream, XmlWriter writer, Guid agentId, Hashtable credentialAssociationList, Hashtable credentialList)

Error Event 3 :

Faulting application name: Microsoft.Mom.Sdk.ServiceHost.exe, version: 10.19.10569.0, time stamp: 0x628d13bb

Faulting module name: KERNELBASE.dll, version: 10.0.14393.5850, time stamp: 0x64253b4c

Exception code: 0xe0434352

Fault offset: 0x0000000000026ea8

Faulting process id: 0x9e8

Faulting application start time: 0x01d9fd266bb55e11

Faulting application path: E:\Program Files\Microsoft System Center\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe

Faulting module path: C:\Windows\System32\KERNELBASE.dll

Report Id: c385e711-042c-4cec-b7e0-153b94785738

Faulting package full name:

Faulting package-relative application ID:

Please advise how to fix this issue.

Thanks,

Adam

Share via

SCOM DB issue

2 additional answers

Your answer