Errors Connecting to the Data Warehouse (Health Service Module Events 31557, 31561, 31563, 31569)
Today I worked with a user...a smart user. He told me a story I'd heard before - "I didn't touch anything but SCOM broke!" - and I have to say this time I believed him....
Basically out of the blue, his management servers (MS) were unable to talk to the DW. And when we looked thru the events below, we noticed that the MS were trying to use the SCOM Action Account instead of the Data Warehouse action accounts. The OpsDB appeared fine, the console ran, and agents were communicating with the management group.
Errors seen on the Management Servers
Log Name: Operations Manager
Source: Health Service Modules
Event ID: 31557
Task Category: Data Warehouse
Level: Error
Description: Failed to obtain synchronization process state information from Data Warehouse database. The operation will be retried.
Exception 'SqlException': Management Group with id ''xxxx'' is not allowed to access Data Warehouse under login ''[ID]''
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode
Instance name: Data Warehouse Synchronization Service
Instance ID: {xxx}
Management group: [name]
+++++++++++++++++
Log Name: Operations Manager
Source: Health Service Modules
Event ID: 31561
Task Category: Data Warehouse
Level: Error
Description: Failed to enumerate (discover) Data Warehouse objects and relationships among them. The operation will be retried.
Exception 'SqlException': Management Group with id ''xxxx'' is not allowed to access Data Warehouse under login ''[name]''
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Discovery.StandardDataSet
Instance name: Alert data set
Instance ID: {xxx}
Management group: [name]
+++++++++++++++++
Log Name: Operations Manager
Source: Health Service Modules
Event ID: 31563
Task Category: Data Warehouse
Level: Error
Description: Failed to enumerate Data Warehouse components for deployment. The operation will be retried.
Exception 'SqlException': Management Group with id 'xxxx'' is not allowed to access Data Warehouse under login ''[name]''
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Component
Instance name: Data Warehouse Synchronization Service
Instance ID: {xxx}
Management group: [name]
+++++++++++++++++
Log Name: Operations Manager
Source: Health Service Modules
Event ID: 31569
Task Category: Data Warehouse
Level: Error
Description: Report deployment process failed to request management pack list from Data Warehouse. The operation will be retried.
Exception 'SqlException': Management Group with id ''xxxx'' is not allowed to access Data Warehouse under login ''[name]''
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Report
Instance name: Data Warehouse Synchronization Service
Instance ID: {xxx}
Management group: [name]
So what's going on?
A little searching pulled up a blog by Stefan Roth - https://stefanroth.net/2012/12/30/scom-2012-event-id-31557-health-service-modules/#comment-26259. I won't duplicate his post, but the gist is that the Data Warehouse profiles had no action accounts associated with them. The solution is to add them back in. Here's what the profiles should look like (at least in my lab). Note: They have "account" in their names but are actually profiles!
Data Warehouse Account (Profile)
Data Warehouse Report Deployment Account (Profile)
How did this happen?
As of now, I'm still investigating...
Comments
- Anonymous
April 01, 2015
Snap with this. Just encountered this in our environment. Can't see anything as to why this occurred. Notes for your use:
- Environment is SCOM 2012 UR5
- SQL backend is clustered The problem manifested itself when we performed a SQL cluster failover. Thanks you for this article, this has been invaluable.
Anonymous
June 08, 2015
Lifesaver. This might have happened when are SQL cluster was rebooted for patching.Anonymous
June 08, 2015
Glad to see I am not the only one that 'did not touch anything and it broke'. I've seen this in two different Operations Manager Shops.Anonymous
May 23, 2016
We experienced this problem when the SQL cluster was failed over for patching.We're running SCOM 2012 R2 RTM on SQL Server 2012 SP2 (11.0.5343)