Sanjeev's System Center Blog

Creating UNIX Diagnostic & Recovery Tasks in SCOM

Sanjeev Gopinath V — Tue, 15 Dec 2015 16:24:28 GMT

Hello All!

Recently, I got an opportunity to work with J Powell from Nine Virtual Technologies. We worked on implementing a diagnostic & recovery solution for some of the critical UNIX workloads in their environment. The challenging part of the problem is that the SCOM 2012 R2 console does not have a UI interface for creating UNIX recoveries & diagnostics. We had to fall back to writing the tasks using MP authoring techniques.

We researched a lot and spent quite some time understanding the various nuances. When our tasks worked, it was a eureka moment for both of us! J & myself thought that it would be a great idea to share the steps with you all. And in no time, J wrote this article to be published in my blog.

Thank you, J for sharing your expertise with the System Center community.

Hope this post helps you save time in creating your UNIX recovery & diagnostic tasks. If you would like to contact J, you can email him at j.powell@ninevirtualtech.com.

Sanjeev

----

Creating Recovery and Diagnostic Tasks for Linux is not as easy as it is for Windows. I knew from creating Recovery and Diagnostic tasks for Windows servers that I could run a command or script. Turns out this is the same for Linux. I can run a script (bash, python, perl, etc) or a single command.

From researching the topic, I could tell that you had to create a custom Management Pack and edit the XML. However, I could not determine how to put the parts together. Also, I could not find anything specific to SCOM 2012 R2. I wanted to provide a write up on how to manage a Linux process and highlight the pitfalls and how to avoid them.

As an example, let’s monitor the sleep process in Linux. If the sleep process stops, I want it to start back automatically and reset the health of the monitor.

First, login to a Linux shell and issue the command ‘sleep 5000’. This will cause the sleep process to run for 5 minutes.

Now, let’s login to the SCOM Admin Console and go to Authoring > Management Pack Templates > UNIX/Linux Process monitoring

Follow the steps below to create the monitor for the sleep process:

1. Right click on UNIX/Linux Process Monitoring and select Add Monitoring Wizard

2. Select UNIX/Linux Process Monitoring and click Next

3. Under Name, Type a description. For this example, we will use Sleep

4. Under Management Pack, select New. This step is very important. Do not add this monitor to any existing Management Packs, as we will need to export and edit this one in later steps.

5. Once you click New, a new wizard will appear. Name the new Management Pack Sleep and click Next.

6. Under Knowledge, you can leave this part blank. Click Create and you will be returned to the previous Unix/Linux Process Monitoring window. Click Next to continue

7. Under process name, click the Select a process button. You will be prompted to browse for a computer. Clicking the Browse button will display all of the Linux/Unix Computers with the SCOM agent installed. Select the one you executed the sleep 5000 command, and hit Select

8. In a few moments, SCOM will display all of the active processes on the Linux server. Select sleep from the list and hit the Select button

9. You will be returned to the Process Monitor window. Notice that the Process Name and Computer to Target fields are now filled in. Also note, the full command is located under the Expression matching Results field. Select it and click Next.

10. On the Configure process monitoring settings, take the default. This will alert us if the sleep process drops to zero. Hit Next

11. On the Summary screen, hit Create.

After a moment, you will see the process named Sleep appear under the UNIX/Linux Process Monitoring section. Now, that we have created a monitor to keep tabs on the sleep process, we need to export the Management Pack. Go to Administration > Management Packs and search for the one you created named Sleep.

Click on Sleep, then click on Export Management Pack in the upper right-hand corner. You will be prompted to save the file. Save it to your desktop.

Now, with a text editor (I use Notepad ++), open the file and let’s go over some of the parts we need to manually edit:

Notice the version highlighted below:

You will have to change this number in order to import the Management Pack again. Let’s change it to 1.0.0.0.

Scrolling down a bit further, you will see a section of References. You will need to add the portion highlighted below:

You likely do not have the Unix Authoring Management pack installed. Download it from this link and install it: https://gallery.technet.microsoft.com/UNIXLinux-Authoring-b16fd2e4

You also need to view the Unix/Linux Authoring MP in a text editor and make certain the Public Key Token and version numbers match up with the section highlighted in the previous screen shot.

Before we get to the Diagnostic and Recovery Task edits, lets take note of one more item. It is the interval in which the monitor evaluates. For this particular one, the default is to run every 5 minutes. For testing, you may want to change this value to 60, which means it will run every minute. Look for the section to edit based on the screen shot below:

Now, on to the edits that will make a Diagnostic or Recovery Task possible.

To help simplify things, in the following sections I will provide four examples: 2 Diagnostic Tasks and 2 Recovery Tasks. The difference between them will be if you are running a command or a shell script. These examples could be cut-n-pasted into your management pack. Be sure not to change the order of items listed or remove any of the parts below, unless otherwise stated. Doing so will cause errors when you try to import the Management Pack.

Diagnostic Task for Executing a Command:

The following should be inserted between the and section. Notice the GUID that follows ProcessTemplate and the monitor are the same. This must match the GUID in the Maintenance $Target/Host/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$ export TERM=vt100; sleep 5000 30 $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$ $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$

At the bottom of the Management Pack, in the section, add the following:


    Run the sleep command

This is how it will look once you reimport the Management Pack:

Diagnostic Task for Executing a Script:

The only two differences between executing a script and a command is substituting the word Command with Script and adding . See the highlighted portion where we changed Command to Script as well as where to add the XML

 
  
    Maintenance
                              
       $Target/Host/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$
       export TERM=vt100; ./sleep.sh
        
       30
       $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$
       $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$

As in the previous example, add the following to the Section:


          Run the sleep command

Recovery Task for Executing a Command:

Recovery Task for Executing a Script:

 
  
      Maintenance
                                
         $Target/Host/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$
	export TERM=vt100; ./sleep.sh
          
	30
         $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$
         $RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$

Now that the edits are complete, save them and let’s import the Management Pack. Be sure to change the version number at the top of XML or it will not import. If it imports successfully, you can kill the sleep task from the Linux shell and you will see a critical error logged in the SCOM console. Use the Health Explorer to view the results of your Diagnostic or Recovery Task.

Troubleshooting Cluster Discovery in Operations Manager

Sanjeev Gopinath V — Wed, 14 Oct 2015 17:06:00 GMT

In this post, we'll be focusing on Cluster Discovery issues. Troubleshooting discovery issues in SCOM is pretty easy when it comes to the day-to-day workflows. All that we do is look through the discovery in the MP – find its data source (VBScript mostly) and try executing it manually. However, with Cluster Discovery the procedure is not that straightforward. Cluster discovery in SCOM is carried out by Cluster Modules that are inbuilt in SCOM. The working of these modules is not exposed like how discovery logics are exposed in management packs. Hence we will have to depend on the events logged in ETL traces. The most common symptom that I hear from my customers is that their SQL instances residing in various cluster instances aren't discovered. While troubleshooting, I’ve seen that it finally ends up being an issue with the cluster discovery. There are 2 types of cluster discovery issues that are common:

There are many cluster instances in the server and only one shows as discovered.
None of the cluster instances in the server is discovered.

(The procedure to check these have been covered below.) Before we go through both of these scenarios, there are a few common troubleshooting steps that we need to follow. Start by checking the following:

Do the nodes that host the cluster have SCOM agents installed?
Do they show up as monitored in SCOM console? (Healthy, Warning or Critical – OK, Greyed out – Not OK)
Do you have the latest Windows Core OS (Base OS) management packs?
Is Agent Proxy enabled for the nodes that host the cluster? (Follow: https://technet.microsoft.com/en-us/library/hh264858.aspx?f=255&MSPPError=-2147217396).

Great! We should be more than set now. Now, open the Agentless Managed view in Administration pane. Look for the cluster we are trying to discover against the list that is displayed in this view. Do you see your cluster here? If not, go back to the previous 4 steps and ensure all of them are completed. If you have just followed the above steps and you don’t see your cluster listed, wait for a few more minutes & try refreshing the view again. If the cluster shows up under Agentless Managed, we need to now check if we can find the individual cluster instances/services. For this, navigate to the Monitoring Pane and choose Discovered Inventory. In the right pane, choose Change Target Type. Look for Windows Cluster Service (For Virtual Server). Click on OK. Now, type in the cluster name and identify how many instances show up. If you find all your cluster services in the list, the above steps did the trick for you. However, if you don’t find one or all of the cluster services, follow the post further. From here, we will divide into two tracks covering each scenario respectively. SCENARIO 1: I see only one cluster instance in the Discovered Inventory (Windows Cluster Service for Virtual Server) view Like I had mentioned, this is a very common scenario and the troubleshooting is not complicated. Try following these steps:

Open SCOM Console and navigate to the Authoring pane.
Select Object Discoveries under Management Pack Objects.
If you find that the view has been scoped already, remove the scope.
Now, look for the following discovery: Windows Clustering Discovery
If you find many, choose the one you see under Windows Cluster or Virtual Server.
Now, right click on the discovery and choose Overrides > Override the Object Discovery.
The next context menu options are left to you. If you want to override it for all the servers, choose the first option. If you would like to have a granular setting, choose an appropriate option that suits.
You will see that there are three overrideable parameters shown and that Multiple Servers Discovery is one of them.
Enable this parameter and change its value from False to True.
Scroll towards the extreme right and check the Enforced option.
Click on Apply & OK.

From here on it could take a few minutes. Grab some coffee and when you come back, you should see the other cluster instances to be discovered as well. If you are looking to see the SQL objects residing in the cluster, you should probably give it some more time. SCENARIO 2: I don’t see any cluster instances showing up in the view To troubleshoot this, we would need to collect ETL traces.

Go to one of the nodes that hosts the cluster and open a cmd prompt in elevated mode.
Type the following commands at the command prompt:

cd C:\Program Files\\server>\tools StopTracing.cmd StartTracing.cmd VER

Restart the Microsoft Monitoring Agent service from Services.msc.
Let the trace run for 15 minutes and then type the following commands:

StopTracing.cmd FormatTracing.cmd

Once the formatting completes, you will find the logs under C:\Windows\Logs\OpsMgrTrace
Look for the file: TracingGUIDSNative.log
Open the log and look for the following line:

[ModulesClusterLibrary][][Error][][CMOMClusterMonitorDiscovery::NotificationImpl] [MOMClusterMonitorDiscovery_cpp292]Some of the Virtual server critical (or key) properties is (are) NULL The above line suggests that the cluster discovery ran but when parsing through the resources, it encountered a number of orphan entries. To resolve the discovery issue, all that we need to do now is clear the Windows Cluster orphan entries from the registry. Hope that helps! Do let me know through the comments below if you ran into any issues when implementing this. - Sanjeev

Monitoring Expiring Certificates using SCOM

Sanjeev Gopinath V — Thu, 17 May 2012 17:00:00 GMT

Managing too many certificates in your Certificate Server? Unable to keep track of when they'd expire? Don't worry! Here's a simple and smart way of proactively monitoring them through SCOM. A completely customizable solution so that you can configure it based on your needs! Sounds exciting? Read further.

Our Strategy:

Our solution for monitoring ceritifcates is based on a script that provides us information about the expiring ceritificates. For each ceritifcate that expires, our script will create an event in the Applications Log. With the help of a SCOM rule, we shall read those events and subsequently notify them as an alert.

The 'Magic' Script

Though there's nothing magic about it, this script is the core part of our solution. It is based on the certutil.exe that is shipped with Windows OS. The certutil provides us an easy way of querying and obtaining information about Certificates. In Windows 2008+ we can export the result set into a CSV which would have made stuff simpler. But, such options are not available in Windows Server 2003. I've designed the script to be compatible for all the OSs. Read through the script, you should be able to get most of it. Don't copy this script from the page. I've pasted it here so that you can have a glance. If you wish to download the script, you can find it at the bottom of this blog post.

' Certificate Expiration Monitoring Script - CertMonitor
' Written by sanjeevgopinath@live.com

' Place the file under C:\CertMonitor\ folder. If you want wish to place it in a different folder, change the dump_file_path variable

' This script is given on an as-is basis without any kind of warranties.
' ##########################
' Type : DAY BASED
' Description : Lists certificates that will expire in the next 1 day.
' ##########################

' ---- VARIABLE DECLARATIONS ----
dim file_obj    ' // Holds the Scripting.FileSystemObject for file handling purposes.
dim line_extract ' // From the file read, we extract line by line
dim cert_count   ' // Holds the current count of certificates found by the script.
dim event_desc   ' // Holds the EventCreate event description
dim file_txt     ' // Contains the File Handle for the Temporary file
'---------------------------------

' -------- INITIALIZATIONS --------
certutil_params="SerialNumber,CommonName,RequestID,RequesterName,NotBefore,NotAfter,CertificateTemplate"
dump_file_path="C:\CertMonitor\Day_Log.txt"
script_type="day"
event_id=101
NewDate = DateAdd("d",1,Date)
' To Add some more parameters add it after 11:59 PM" like this --> & "YOUR CONDITION" & """"
restrict_param=" -restrict ""NotAfter>=" & NewDate & " 12:00 AM,NotAfter<" & NewDate & " 11:59 PM" & """"
' Creating Object to work with Shell
Set WshShell = CreateObject("WScript.Shell")
' Running the Certutil command and dumping the output in dump_file_path
cmd_complete="cmd /C certutil -view " & restrict_param & " -out """ & certutil_params & """ > " & dump_file_path
WScript.Echo cmd_complete
cmd_status = WshShell.Run(cmd_complete, 1, true)
' Logging event for the start of script execution.
Set cmd_status = WshShell.Exec("eventcreate /t information /l application /id 100 /d ""Running Certutil -view for " & script_type & " based rule.This is just an informational alert generated by the Certificate Expiry Monitoring Script."" /so ""CertMonitor""")
' Initializing the file handlers
Set file_obj = CreateObject("Scripting.FileSystemObject")
Set file_txt = file_obj.OpenTextFile(dump_file_path,1,False)
' Low Index contains the Certificate Details starting position from the top.
low_index = 0
' High Index holds the position of end of Certificate details.
high_index = 0
' At any given point, cert count tells how many certificates the script has parsed so far.
cert_count = 0
' Initializing the event description
event_desc = "Certificate Details : "
' Sub Str is a temporary string used for trimming and removing the quotes
sub_str = ""
' ------------------------------------

' --- LOGIC STARTS HERE ---------------
' Extracting line by line using split. chr(10) represents newline.
line_extract = Split(file_txt.ReadAll(), chr(10))
' For every line in the dump file
For i = 0 To UBound(line_extract)
' If the line is equal to a Carriage Return, it may represent end of Certificate.
If line_extract(i) = VbCr Then
  ' The initial part of CertUtil contains information about the fields. We ignore it by not counting the first occurence
  If cert_count > 0 Then
   ' This loop extracts all the certificate information and concatenates it in event_desc
   ' low_index + 1 is used so that we remove the "ROW #" line
   For j = low_index + 1 To high_index
    ' This removes unnecessary spaces.
    sub_str = trim(line_extract(j))
    ' This removes all the quotes so that the quotes aren't closed by themselves. For e.g. in EventCreate
    sub_str = Replace(sub_str, """", "")
    ' The last character is a new line / NULL character. If that's present, we won't get full event description.
    event_desc = event_desc + "" + Left(sub_str,Len(sub_str)-1)
   Next
   event_desc = event_desc + "This event was generated by the Certificate Expiry Monitoring Script."
   ' Logs the event in the Application Log
   Set event_return = WshShell.Exec("eventcreate /t error /l application /id " & event_id & " /so CertMonitor /d """ & event_desc & """")
   ' Waiting for the Command to complete
   Do While event_return.Status = 0
    WScript.Sleep 100
    WScript.StdOut.Write(event_return.StdOut.ReadAll())
    WScript.StdErr.Write(event_return.StdErr.ReadAll())
   Loop
   ' Reinitializing event_desc so that we don't have the previous values.
   event_desc = "Certificate Details : "
  End If
  ' Incrementing the low_index to point to the starting of next certificate.
  low_index = i + 1
  ' Incrementing cert_count value as we parsed one paragraph (one para is imagined to contain cert details) completely.
  cert_count = cert_count + 1
Else
  ' If we find the summary information, we stop the script then and there.
  If instr(1,line_extract(i),"Row Index",0) > 0 Then Exit For
End If
' Incrementing as we go line by line
high_index = high_index + 1
Next

How does it work?

The script is expected to run once everyday. To confirm if it ran, you can check the Application log for 100 EventId. When the script runs, it queries for all the expiring certificates and puts the result it in a temporary file. Since the first part of the result is junk, we omit it and just take the part where we have information about expiring certs. For each certificate entry, it creates a 101 event in the Application Log. If you want to change the event id, you can change the value specified for the event_id variable. Also, in this script we just query few parameters like serial number, etc. If you want to delete or add few more parameters, you can change the certutil_params variable.

Say, if you want to extend monitoring and be notified before a week or a month, you can do the following...

Change the value 1 in NewDate = DateAdd("d",1,Date) to 7 if you want to be notified before a week and 30 if you want to be notified before a month.

You can also change the event id when you wish to monitor for week and month. This way, you can configure the rules to alert with respective priorities. For example Day = High, Week = Medium.

What is required from SCOM side?

Create an alert generating rule to monitor for NT Events 100 and 101 (or the event Id you have specified). To be very precise, you can specify that the Event Source is 'CertMonitor'.

If you want to know how to create a alert generating rule for monitoring events from the NT Event Log, read this : https://technet.microsoft.com/en-us/library/bb309568

As the rule may run on all Windows Computers, restrict it to run only on the cert servers or the servers where you need to monitor expiring certificates.

How do I schedule the script?

You can use Windows Task Scheduler for scheduling the script to run once everyday. For more info : https://technet.microsoft.com/en-us/library/cc748993

day.vbs