You will find a lot of hopefully useful information related to SCOM, OMS, Azure Insights, PowerShell and SMA in this new blog.

So stay tuned and bookmark our new blog: https://blogs.technet.microsoft.com/germanageability

Cheers,

Dirk

Today I want to talk about an issue I bet almost every SCOM Admins who uses X.509 Certificates for authentication had stumbled upon and that I experience quite often in the field:
The Certificate used for SCOM authentication has been changed and you forgot to reimport this Certificate into SCOM with momcertimport.exe.

Most of you might say: Oh man, that’s old hat! That’s why we have e.g. Rapahael Burri’s excellent PKI MP for.
And yes, you are absolutely right! But a Certificate monitoring MP only warns you if a Certificate is about to expire or has already expired, but it usually does not warn you that the SCOM health service still uses a different Certificate!

Scenario

Let me give you an example:
The SCOM Management Server and Gateways are using Certs for authentication. The Cert is about to expire (that can be monitored e.g. with Raphael’s PKI MP) and will be updated either manually or automatically (e.g. via AD CA auto-enrollment).
Changing Certs automatically is very efficient and lean, but can be extremely dangerous to SCOM because:

Updating a Cert usually means changing the serial number of the Cert (e.g. from SNR 1 to SNR 2). So the serial number of the Cert in the Certificate store of the local computer and the serial number of the Cert used by the SCOM health service (stored in HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Machine Settings\ChannelCertificateSerialNumber) does not match anymore.
In the screenshot below the serial number in the registry does not match the one on the Cert (I deleted the first byte):

Will updating the Cert cause any immediate trouble? Not necessarily. Everything will continue to work as long as you do not restart the SCOM services, because SCOM caches the specified Cert at startup time. Once you restart the services SCOM will notice that the Cert with the serial number 1 is not available anymore. SCOM will then write an Error Event and immediately closes all connections authenticated by the Cert (mainly the Gateways).

This scenario (SCOM Cert has been changed, everything is running but SCOM uses an invalid cached Cert) is unfortunately not covered by any Management Pack known to me.

A possible solution

To get an actionable alert in case that this scenario happens I wrote a small demo MP with only one monitor that does this job:

• It runs on all Management Servers (incl. Gateways)
• Monitor affects health state of the Windows Computer (to get a visible state change as well)  
• Monitor checks on a regular basis (e.g. every 1h) if the serial number found in HKLM can be found in the Certificate store of the local computer. If not it will create an actionable alert for the operations team.
• I know, that this might not be the best way, because the Cert in the store might be invalid. But invalid Certs should already be monitored by the PKI MP.

Monitor in an error state (mismatch will be detected, serial number will be shown and the Operator gets the hint to reimport the Cert with momcertimport.exe):

Monitor in a healthy state:

You can find the unsealed MP attached to this post.

Summary

Even if you monitor your Certs closely you might run into the situation that you update a SCOM Cert but forget to register this new one in SCOM as well. My demo MP monitors this situation and creates an actionable alert. Based on this alert a SCOM Admin should then take appropriate action and import the new Cert into SCOM with the help of momcertimport.exe.

Custom.SCOM.CertCheck.xml

The attached function test-customopsmgrtracerunning

• does not require any parameter
• returns $NULL if it cannot determine the Tools directory on an Agent or Management Server
• returns $True if it finds loaded OpsMgrTrace ETL files -> Trace is running
• returns $False if it does not find loaded OpsMgrTrace ETL files -> Trace is not running

The function in action:

test-customopsmgrtracerunning.ps1

Use case

Working with Overrides for SCOM workflows is a common task. There is a lot of documentation available describing how to set Overrides for specific targets and how to store them correctly (preferably in separate custom Override MPs).
But I figured out that many customer still struggle with the handling of their Overrides. Common questions I often hear are:

• Who has set a specific Override?
• Who was the client/customer who ordered an override (e.g. a specific user or department)
• What is this Override doing? Why was it set?
• How can I handle temporary Overrides which are only necessary for a certain time period?
• In ITIL driven environments: what is the change ticket id for this specific override?

Out of the box SCOM provides the “Overrides” Report which lists all Overrides for specific MPs created in a specific time frame. This is really useful as a general overview but does not answer all questions asked above.
Well, each Override has a description field which can be edited with the required information. Simple, easy, end of story.

Hmm, yes but there are still some caveats:

• What if I want to report on the entered data in an automated and structured way?
• The description field is not a mandatory field. How can I force my Operator to enter the required information in a structured way?

How I do it: Solution Overview

My solution for handling Overrides consists of two parts:

1. Structured naming convention for each Override description in form of a written policy/work instruction or however you might call it.
2. A Management Pack which reports and controls the compliance of the defined naming convention

1. Structured naming convention
What kind of convention you will use and which kind of delimiter you choose is completely up to you. I will usually use this syntax:

<Creation date>#<Author>#<Client>#<Expiration date>|<NULL>#<Ticket ID>|<NULL>#<Description>

Where:

• Creation date: YYYYMMDD
• Author: Username or any other common abbreviation of the user who sets the Override
• Client: Username, department name or SELF, describing who has ordered the Override. SELF it Author and Client are the same person.
• [Optional] Expiration date for temporary Overrides: YYYYMMDD or NULL/empty string.
• [Optional] Ticket ID of ITSM tool e.g. for the corresponding change. Can also be NULL/empty
• Description: Textual description why the override was set

Examples:

• 20151001#Dirk#SELF###Set because … 
  Override was set and ordered by Dirk on October 1st 2015 without a ticket and it is never expiring
• 20151015#Dirk#John##SR123456#Set because …
  Override was set by Dirk on October 15th  2015 on behalf of John with ticket id SR123456 and it is never expiring.
• 20151018#Dirk#John#20151130##Set because …
  Override was set by Dirk on October 18th 2015 on behalf of John. This Override will expire on November 30th 2015.

2. Management Pack
The attached simple demo Management Pack consists of one custom data source which

• Executes a custom PowerShell script on a daily basis (SyncTime 9:00, Intervall 86400 seconds), which
  • Creates an export of all existing Overrides as CSV for later analysis and reporting
  • Checks if there are Overrides in Management Packs using a specific prefix (in this demo XXX) not in compliance with the above defined naming convention and creates a specific property bag with the result
  • Checks if there are temporary Overrides in Management Packs using a specific prefix (in this demo XXX) which have already expired and creates a specific property bag with the result
• A custom unit monitor type which consumes the property bags
• One unit monitor which creates an alarm if there are Overrides not compliant with the naming convention
• One unit monitor which creates an alarm if there are expired temporary Overrides

Explanations:

• MP Filter in the script
  I will not check the syntax in ALL Override MPs, because there might be default overrides in certain MPs which do not comply with the naming convention for a good reason. As it is best practice to store all custom Overrides in own custom Override MPs, these MPs should use a certain common prefix. In this demo I have chosen XXX as the prefix. So only Overrides contained in MPs beginning with the internal name XXX will be checked.
• Workflow target
  I have chosen the RMS Emulator role as a target for all workflows. This might not be the best choice, but it is a good trade-off between simplicity, functionality and scalability.

The solution in action

A correct Override description according to the naming convention would look like that:

But this is just a convention which cannot be enforced...
Now the Override Handling Management Pack enters the stage :)

After importing the MP you can see three Monitors attached to the Configuration Health roll up Monitor on the RMS Emulator:


If you create a new Override in a MP starting with XXX and not complying with the naming convention for the description field by e.g. leaving it empty…

… or by creating a temporary Override which has already expired …

… these monitors will change state and generate an alert with the problematic Override GUIDs included in the alert description:

Outcome 1: Detailed alarming
So you will get a detailed monitoring and alerting thus enforcing the naming convention and never forget any expired Overrides anymore!

The data source will also create an export of ALL available Overrides as a CSV file with the name YYYY-MM-DD_<Management Group>_OverrideExport.csv:

This CSV file contains a lot of information about the Overrides. Once you import it into Excel it could look like this (I have filtered the table to match the three previously created Overrides and filtered out all unnecessary columns):

Outcome 2: Flexible reporting
You can use the daily CSV files to create flexible reports in Excel and to track changes!

The PowerShell Script in the data source will also generate detailed Event log entries (9001-9099) in the Operations Manager Eventlog on the RMS Emulator which simplifies troubleshooting and debugging:

Implementing the solution in your own SCOM environment

The attached unsealed Management Pack is for demonstration purpose in TEST environments  only and not suited for production environments.

The requirements for testing this MP:

• Management Server Action account must be member of SCOM Admin role. The PowerShell data source will be executed under this account and will try to connect to the SCOM SDK. If you don’t like this, simply modify the MP by creating a suitable RunAs account to execute the data source under a different security context.
• Before you import the MP please modify the overrideable parameters PrefixCustomOverrideMP and ExportDirectory for both Unit Monitors to suit your needs or create Overrides.
  Please be aware that you should set the same values in both workflows to make use of Workflow Cookdown!
  
  

Conclusion

That is my way of handling SCOM Overrides. I use the naming conventions and the automation processes behind in several customer environments successfully. It is simple to implement and can easily adapted to your specific needs.

Let me know what you think of it.

Community.OverrideHandling.xml

SQL Server Visualization Library (dashboards) features and changes in v6.6.3.0

• Implemented batching to all data aggregation mechanisms to ensure low temp database space and log space usage.
  • Please see Notes section below for further details.
• Implemented a control bit to divide group from non-group references to save space in aggregated data storage.
• Fixed error message in SQL DB Discovery script that erroneously stated "… SQL Agent Job discovery script … failed".
• Improved performance!
• Dashboard displays behave in line with user permissions:
  • The user can see the groups according to his/her access level only.
  • Read-Only mode for non-authors provides basic functions only: navigation, changing of the personalization settings.
• Dark, Light, and Contrast themes have been added.

IMPORTANT NOTE (22.10.2015): SQL Product Group has been removed the MP version 6.6.2.0 from download. Read more here.
If you are affected by the issues caused by the MP you can mitigate the issue by executing this SQL code on your OpsMgrDB according to the Product Group

alter procedure sdk.Microsoft_SQLServer_Visualization_Library_UpdateHierarchy as RETURN 0;
GO
alter procedure sdk.Microsoft_SQLServer_Visualization_Library_UpdateLastValues as RETURN 0;
GO
update sdk.Microsoft_SQLServer_Visualization_Library_Tables set LastProcessedId = 9223372036854775807;

IMPORTANT NOTE (20.10.2015): Please be careful with this release of the MP and test it thoroughly. There might be issues with the SQL 2014 MP and fast growing Log files which I did not notice in my test environments so far but other customers have. Thanks to Vladimir and Patrick for pointing me to this issue.

This is just a short info that  a new SQL MP is available for download here.

According to the MP guide it contains several fixes:

• Added a support for disabled TCP/IP protocol
• Fixed performance metrics error that may occur on some localized versions of Windows
• Fixed bugs in monitor tiles on SQL Server Summary Dashboard
• Fixed incorrect performance of Transaction log free space monitor
• Added new type of events from failed discoveries; added a new rule that collects such events
• Added overrides to prevent various scripts timeout failure
• Removed some 1X1 tiles from Summary Dashboards
• Filestream filegroups are excluded from discovery for now
• 2008/2012 Summary Dashboards tiles were reorganized

In my test environment everything seems to work now, so happy testing and evaluating :)

SCOM alert handling scenarios

What kind of alerts do we have to handle in SCOM? What are good practices and what are bad practices?
I came along with five different alert handling scenarios (see attached PDF for better readability):

Automating the scenarios

In an ideal world:

• We only get actionable, meaningful alerts -> every alert makes perfect sense
• every monitor alert gets automatically resolved once the underlying root cause has been fixed
• no SCOM Operator will ever close a monitor alert without ensuring that the causing monitor is healthy

TO BE VERY CLEAR:
IF YOU CLOSE AN ALERT CREATED BY A UNIT MONITOR THIS MONITOR WILL NOT SWITCH BACK TO A HEALTHY STATE. AS LONG AS THE ROOT CAUSE HAS NOT BEEN FIXED THE MONITOR STAYS UNHEALTHY AND WILL NOT TRIGGER ANY NEW ALERT!!!

But unfortunately we don’t live in an ideal world so I created a little script (you can download it from TechNet Gallery here) to help me with the alert handling and management according to Microsofts good practices.
This script handles alerts:

• It closes all open SCOM alerts (ResolutionState <> 255) where the TimeRaised property is older than a certain age.
  -> This will clean up my console if the Operators have forgotten to close alerts in time
• It closes all new SCOM alerts (ResolutionState = 0) where the TimeRaised property is older than the configured SCOM alert grooming age (default 7 days)
  -> This will clean up my console if the Operators have forgotten to acknowledge or otherwise handle open alerts
• It closes alerts created by specific workflows in a configurable amount of time by using an XML file with custom alert handling rules
  -> This allows you to configure exceptions to the rules above. E.g. some alerts created by the AD MP rules are not relevant after X hours. So you can close the alert safely because it will be triggered again if the issue still exists.
  

But this script takes also care of the causing monitors and their health state:

• If a monitor alert is closed by the script it will automatically reset the causing monitor if necessary.
  -> This will ensure that if the root cause has not been fixed the monitor will trigger a new alert during the next execution cycle.
• It analysis closed monitoring alerts for the last 24 hours. If there is a closed monitor alert not tagged by the script than it is probable that an Operator has closed this alert manually. The script checks if the causing monitor has been reset. If not, the script will reset the monitor.
  -> This handles the common mistake that Operators will close alerts without taking care of the causing monitor.

The script is a working proof of concept. Requirements are:

• PowerShell 2.0
• SCOM 2012 PS module (e.g. SCOM 2012+ Console installed)
• SCOM 2012 agent installed

The reason for the SCOM agent requirement is that you can easily execute this script in a SCOM workflow context (e.g. timed rule). But you can also execute it simply in a shell. It has a detailed command based help included which explains all parameters:

The script will write detailed log events to the Operations Manager event log (hence the requirement for the SCOM 2012 agent):

What do you think? Do you have more SCOM alert scenarios that should be handled? Please let me know! Feedback is als always highly appreciated.

SCOM-Alert-Management-Scenarios.pdf

Ok, it was not rocket science, but much harder as I thought. Quickly I came up with a function which used Get-SCOMRunAsProfile and a method of the Management Group called GetMonitoringSecureDataHealthServiceReferenceBySecureReferenceId() to process all RunAsAccounts and list there usage in profiles. Unfortunately this method only lists accounts in profiles where the target is the HealthService object itself. If you use other targets (classes, instances) it won’t work.

Hmm, looking at the console I found a link “Where is the credential used?”, which does the trick.

But how do I translate this into PowerShell?
Thanks to our highly knowledgeable and extremely helpful Support Engineer Mihai Sarbulescu who showed me the code behind this link I could finally finish this task!

get-customscomrunasaccounttoprofile

So let’s see the function in action:
I have several RunAsAccounts in my Lab environment…

and e.g. a demo Profile with several accounts associated with different targets:

When you run the function get-customscomrunasaccounttoprofile it will list all SCOM RunAsAccounts and their usage like this:

The function

• lists all RunAsAccounts
• shows their usage in profiles and
• displays the corresponding target ID and the display dame of the target

You can download the most current version of the function at TechnetGallery.

Overview (Datacenter) Dashboard

You can create overview dashboards for SCOM groups and include State and Alert data for the group as well as aggregated performance and monitoring data:

 
and

Individual (Instance) Dashboards

The datacenter dashboards allows you to drill into your data and create additional tiles with monitoring and performance data:

And the best thing is:
You can use these dashboards for all other (custom) SCOM classes, counters and monitors!

So you are not limited to SQL data. This is just a quick demo dashboard for one of my test Management Groups:

You can find all necessary details and steps to set these dashboards up and running in the attached SQLServerDashboards.docx document which is also included in the new SQL MP.

Call to action

Download and install the new SQL MP as soon as possible in your favorite test environment. Read the documentation and quickly create powerful, great looking Dashboards!

SQLServerDashboards.docx

Credits for this post
All credits and knowledge for this post goes completely to Anton Gritsenko for his great post here and to Mihai Sarbulescu for his excellent post here.
I have just summed the topic up in a way that I (and hopefully all other interested people) can understand and hopefully remember this topic.

Why do I need to know this stuff?

Good question. Knowing this stuff answers questions which you might have been asked yourself already:

• Do you ever wanted to know, why a MP produces different GUIDs in a Lab (where it is unsealed) and a production environment (where it is sealed)?
• What happens when you use GUIDs for instances or classes in a MP and move the MP between Management Groups (LAB -> Test -> Production)?

Demo XML

For demonstration purposes I created a quick and dirty MP with Silects MP Author (see MP attached). It contains two classes, a discovery and two monitors:

“Static” MP elements vs. “dynamic” instances

When we talk about GUIDs we can roughly distinguish into two main classes of GUIDs:

• “static” MP Elements
  like the GUID for the MP itself, classes, monitors, rules etc. and
• “dynamic” (Class) Instances
  like a disk object or a health service object and so on

How are GUIDs for MP elements created?

GUIDs for MP elements are created by using two SQL functions in the OperationsManager db:

• dbo.fn_MPId for Management Packs
  Example sealed MP: SELECT dbo.fn_MPId('MY.MP.NAME, '%PUBLICKEY%')
  Example unsealed MP: SELECT dbo.fn_MPId('MY.MP.NAME, NULL)
• dbo.fn_MPObjectId for other Management Pack elements like classes
  Example sealed MP: SELECT dbo.fn_MPObjectId('MY.MP.NAME,'%PUBLICKEY%',‘MY.NEW.CLASS')
  Example unsealed MP: SELECT dbo.fn_MPObjectId('MY.MP.NAME, NULL, 'MY.NEW.CLASS')

When you have a look at the function code you can see, that SCOM combines the function parameter and uses a SHA1 hash function to calculate the GUID:
 

%PUBLICKEY% is the public key (aka KeyToken) of a sealed MP. You can always get this key for any sealed MP by using the get-scommanagementpack Cmdlet.

It is important to note here, that both SQL functions require all uppercase letters, so don’t write e.g. 'My.MP.Name'!

Ok, let’s verify this on an example:
Without importing the MP into a Management Group I check the GUID via PowerShell:


And now let’s verify it with the correct SQL function:

YESS, right GUID.

Now let’s import the MP and verify a class. Again first via PowerShell:

And again with the correct SQL function:

Perfect match!

How are GUIDs for MP instances created?

Ok, what about GUIDs for instances? When we talk about instances we have to deal with two different types of instances:

• Singleton class instances  (e.g. Groups) and
• Non-singleton class instances (e.g. Disk objects)

Singleton class instances will be treated like all other classes and will use the function dbo.fn_MPObjectId('MY.MP.NAME, '%PUBLICKEY%', ‘MY.NEW.SINGLETONCLASS ')

Non-singleton class instances will be treated differently and in a much more complex way:

StringToHash = ‘TypeId={<TypeGuid>}[,{<KeyPropertyId>}=<KeyValue>][,{<KeyPropertyId_2>}=<KeyValue_2> ...]’

The string to be hashed will be calculated by combining:

• the GUID of the target class (TypeID) and
• all key properties in a way like this:
  {GUID of key property 1}=VALUE_OF_KEY_PROPERTY_IN_UPPERCASE_LETTERS

Sounds complicated? It IS complicated unfortunately. Let’s see an example and start with my demo class which contains only one instance and start again by using PowerShell:

My SCOM.DEMO.GUID.SCOMDEMO.Application class is a child of Microsoft.Windows.LocalApplication. We have to find the key property which can be a bit tricky. In my case, the key property is PrincipalName (5c324096-d928-76db-e9e7-e629dcc261b1) from class Microsoft.Windows.Computer (EA99500D-8D52-FC52-B5A5-10DCD1E9D2BD).

Using Mihai’s SQL function it will look like this:

And again: a perfect match!

Conclusions

Conclusion Nr 1:
MP element GUIDs are calculated based on their Name (MP name, element name) and, if sealed, on the KeyToken of the MP. As long as you do not change one of these properties (e.g. use a new MP seal key), the GUIDs will always be the same, regardless of the Management Group the MP will be imported into.

Conclusion Nr 2:
As soon as you seal (or unseal) a Management Pack and import it into another Management Group, the GUIDs will change, because the KeyToken is one element in the hash function used by the SQL functions.

Conclusion Nr 3:
As long as the key property value won’t change (which his highly unlikely) and we are using sealed MPs the GUID for a class instance is also predictable.

I hope, that this post will shed some light on the creation and calculation of SCOM GUIDs. The same mechanisms are also valid for Service Manager (SCSM).

SCOM.DEMO.GUID.xml

CAUTION and NOTE

1. This script is a proof of concept to demonstrate how to automate password changes in SCOM. It is neither fully tested nor supported by me or Microsoft in any way. Do not use this script without analyzing the code carefully and maybe adapting it to your needs. Use it at your own risk!
2. There are always several way to perform certain tasks. Some are much more elegant than others. I do not claim to always use the most elegant ways in my script *smile* 

Ok, let’s get started:

Script help
The script is documented by a command based help:
 

Script call
Calling the script with random individual passwords (default setting) and backing up the SRS encryption key:
PS> SCOM-SEC-ChangeSVCPassword.ps1 -managementserver myms.domain.com –BackupSRSEncryptionKey

Script consent
To prevent any accidental use of the script:

Account verification
After testing the environment and loading the account information the script will present the information to you and ask for your consent:

The whole picture
If everything goes fine, you will not get any further consent prompts and the script shell output will look something like that:

Generated (log) files
The script will generate three or four files:

where

• YYYY-MM-dd_hh-mm_OpsMgr_SVC_PasswordChange.log
  contains detailed logging information (without any password information)
  
• YYYY-MM-dd_hh-mm_OpsMgr_SVC_PasswordChange.csv
  contains all account information including the new passwords in clear text
• YYYY-MM-dd_hh-mm_OpsMgr_SVC_PasswordChange_Error.log
  contains potential PowerShell errors.
• [optional] YYYY-MM-dd_hh-mm_[SRS Servername]_SRS_EncryptionKey.snk
  contains the SRS encryption key. The password for this key is the password of one of the accounts. Please check the YYYY-MM-dd_hh-mm_OpsMgr_SVC_PasswordChange.log for details

Modifed account description
Each AD account gets a description for better documentation appended:

This is also true for RunAs accounts:

Testing and verification

Do not forget to test every single SCOM component after the change! Verify that everything is up and running. If you have 3rd party software which requires password changes as well, please do this now!

Conclusion

By using an automated approach like the demonstrated script

• I can change SCOM service account passwords even in complex environments in under 2 minutes
• Because it is fast it allows more frequent password changes, thus increasing the overall security of your environment
• All steps and actions taken are documented and logged automatically

I hope you found this blog series interesting and it encourages you to create your own automation tools for your SCOM environment(s).
As always I would love to hear your feedback and comments.