cchamp's opsmgr blog

Loading the SCOM PowerShell Module, For Real This Time!

cchamp — Wed, 30 Sep 2015 08:29:47 GMT

This is just a quick post and reference for how I have come to load the SCOM PowerShell module for scripts that requiring SCOM cmdlets. It evolved from several examples I borrowed (thank you to those sources which I've since lost), combined with some real world experiences (such as running inside a SCOM rule / PowerShell Workspace API, running from Orchestrator, running from a scheduled task, running from a machine with the console and PS module install but not a MS), and is now:

$OMCmdletsTest = (Get-Module|% {$_.Name}) -Join ' '
If (!$OMCmdletsTest.Contains('OperationsManager')) {
    $ModuleFound = $false
    $SetupKeys = @('HKLM:\Software\Microsoft\Microsoft Operations Manager\3.0\Setup',       
        'HKLM:\SOFTWARE\Microsoft\System Center Operations Manager\12\Setup')
    foreach($setupKey in $SetupKeys) {
        If ((Test-Path $setupKey) -and ($ModuleFound -eq $false)) {
            $setupKey = Get-Item -Path $setupKey           
            $installDirectory = $setupKey.GetValue('InstallDirectory')
            $psmPath = $installdirectory + '\Powershell\OperationsManager\OperationsManager.psm1'
            If (Test-Path $psmPath) {
                $ModuleFound = $true
            }
        }
    }
    If ($ModuleFound) {
        Import-Module $psmPath
    } else {
        Import-Module OperationsManager
    }
}

Multi-home Management Group Automation MP

cchamp — Tue, 08 Sep 2015 22:15:04 GMT

Hello, just fulfilling a quick ask with this one. With SCOM 2012 we were given the ability to manage agent Management Group configuration using the Agent Config Manager Object or AgentConfigManager.MgmtSvcCfg. It can be leveraged in VBS or PowerShell to Add, Remove, and List SCOM management groups configured on the agent. This isn’t groundbreaking news and the methods are well documented, and plenty of community examples exists already.

That said, I see it asked frequently about what the best method of running a script to modify MG settings against their agent install base. SCCM and remote PowerShell are common solutions, but with this last go-around, I thought why not leverage SCOM itself (assuming it’s not an all-out removal).

Below is a link to a VSAE project for a Management Pack that contains two disabled rules and a script-based data source (VBS) to help automate the addition and removal of management groups for agents. The rules can be overridden and enabled to Add a MG w/a Management Server, or remove a MG.

I’ve done my best to include logging and protections against bad things (like removing the last MG on an agent), but please be cautious as this MP could be dangerous and easily break every agent. As always:

This is provided AS-IS, as an example only, and contains no warranty. The MP should not be used in a production environment without careful review and planning.

Have fun!

https://github.com/cchamp-msft/MultiHome.ManagementGroup.Automator

Sky Blue–How to keep monitors from going unnoticed after closing an alert

cchamp — Mon, 17 Aug 2015 22:41:18 GMT

Scenario: A 3-state logical disk monitor targets a disk that meets the warning threshold. A warning alert is raised.

The Operations Team sees the alerts, consults SOP guidance which states ignore warnings, and manually closes the alert. The alert is gone, but the underlying monitor is still unhealthy with a state of warning. A day later the disk space used is now critical, and the monitor state changes to match.

The problem is the alert is closed and it won’t fire again until the monitor is healthy.

The critical alert is closed. Operations doesn’t look at closed alerts. And so the disk reaches capacity, potentially bad things happen, and ultimately someone asks, “Why didn’t we get an alert?”

Well we did, and we closed it.

Let’s consider how alerts are generated from monitors. We often speak in terms of an alert being generated when a threshold is met, and in the case of monitors that over simplifies the process and leaves out an important step. The thresholds trigger a condition detection path that leads to a state change. That state change can then prompt for an alert to be generated. In the case of 3-state monitors, we only generate one alert. When we change from a warning to a critical state, alert severity can be (and often is) configured to match the monitor state. The not so intuitive part of this model reveals itself in that when an operator sees a non-actionable alert, the default behavior is to close the alert. Closing the alert does not affect the state of the monitor. That’s when we find ourselves in situations where the state of a monitor can be warning or critical, but we have no corresponding alert. The state changed so long ago, the alert has been groomed after someone closed it.

So how do we avoid these situations?

Well, first we can always submit product feedback, such as something like:

https://systemcenterom.uservoice.com/forums/293064-general-operations-manager-feedback/suggestions/9356712-a-way-to-reset-monitors-automatically-when-alert-I

But until then we can address the behavior with a PowerShell driven Management Pack I created called Sky Blue. Thanks to Peter Lem for the name, a play on Green Machine and the fact that management asking about “missing alerts” always feels like a rainy day.

Now the ask in the above feedback link is somewhat different than an ask I had from a customer which originally drove this workaround. Instead of providing a task or a way to replicate Green Machine’s resetting of monitor states, the approach with Sky Blue is to enforce good operator behavior in the SCOM console. Sky Blue protects against accidental alert closures by re-opening closed alerts where the monitor state is not healthy. This is done via a rule that runs every five minutes (override-able) against the All MS Resource Pool. The rule runs a PowerShell script that queries for all closed alerts within the past two intervals that were generated by a monitor, checks the state of those monitors, and re-opens (sets state to 0) any alert where the monitor state is not healthy and not in maintenance mode.

Or more specifically, albeit condensed:

$interval=300
$d=(Get-Date).ToUniversalTime().AddSeconds($interval*-2)
$alerts=(Get-SCOMAlert -Criteria "ResolutionState = 255 And LastModified > '$d' And IsMonitorAlert = 'True' And MonitoringObjectHealthState <> 1" | Sort-Object -Descending TimeRaised)
foreach ($alert in $alerts) {
    $alertCheck = Get-SCOMAlert -Criteria "ResolutionState <> 255 And IsMonitorAlert = 'True' And RuleId = '$($alert.RuleId)' And MonitoringObjectId = '$($alert.MonitoringObjectId)'"
    if ($alertCheck.Count -eq 0) {
        $alert.ResolutionState = 0
        $alert.Update('Sky Blue found a closed alert from a monitor that is not healthy. Changing Resolution State to New.')
    }
}

Better yet, head over to GitHub for the whole VSAE project with the PowerShell wrapped up in a rule with a scheduler. Pull Requests and feedback welcome.
https://github.com/cchamp-msft/SkyBlue

Or if you want an unsealed copy of the built MP:
https://github.com/cchamp-msft/SkyBlue/blob/master/SkyBlue/bin/Release/SkyBlue.xml

The configuration of the rule is as follows:

IntervalSeconds – Default 300 – How often to run the rule (300 is fairly aggressive, 3600 is still effective).
NoReopenSeconds – Default 0 – Useful for sub 60 second interval so that it won’t re-open anything that auto-closed before the monitor has fully reconciled. This setting is not for production use, but if I recall correctly I set this to 30 when testing on a 30 second interval.
TimeoutSeconds – Default 60 – Timeout for the PowerShell script, 60 should suffice.
GenerateAlert – Default true – Whether or not to generate an alert on error condition
Debug – Default false – Whether or not to make a mess of the Operations Manager event log on the MS running the rule in the All MS Resource Pool.

Please note this is provided as-is and contains no warranty. Thank you Michael Bullwinkle for kindly reviewing and editing this post.

Authoring with Azure Run As Certificate Credentials

cchamp — Thu, 16 Apr 2015 22:07:00 GMT

As Azure becomes more prominent, I’m running into more asks on how monitor Azure from an on-premises perspective. As of this writing, we have a couple management packs available now that can help with this. One that monitors cloud services (VMs and storage), and it also discovers the presence of SQL Azure—System Center Management Pack for Windows Azure v1.1.238.0, 10/2/2014. And another for specifically monitoring SQL Azure databases—Windows Azure SQL Database Management Pack v1.5.4.0, 5/7/2013. There’s an older one out there as well, but that’s out of the scope of this post. Those are great, but they don’t cover everything as of yet. In the meantime, as per the guide for the Windows Azure MP says, any other gaps in monitoring need to be addressed by “[using] MP Authoring tools to create additional collection rules and monitors.” Sounds trivial enough…

When considering what’s involved with monitoring Azure, there are two major requirements: How to connect to Azure, and how to authenticate to Azure. The way the Windows Azure MP deals with the first is through a custom resource DLL included in the MPB file. The DLL contains functions that the MP leverages via Data Sources to feed the discoveries and monitoring workflows. While this is great for what’s provided and likely offers better performance at scale, it doesn’t offer much in terms of extensibility for Azure features outside the scope of the DLL. For any custom monitoring, I much rather rely on Azure PowerShell. If you’re already acquainted, great! If not, check it out, connect it to your subscription and poke around.

The first step in working with Azure PowerShell highlights the the other hurdle—authentication. Azure PowerShell typically uses a subscription registered with Add-AzureAccount, an interactive process that presents a windowed browser for logging in. Since interactive isn’t choice for automating or embedding in a terminal-less workflow, Set-AzureSubscription gives up a better method for automating the authentication to Azure using a management certificate. And so the sub-challenge of where to keep the management certificate presents itself. It is entirely possible to manually (or automatically for matter) deploy the certificate to the Personal Store, and then hardcode the private key in a custom management pack. Or, we can re-purpose…

The Windows Azure MP manages authentication through three Run As Profiles that it’s able to configure based on the responses entered into the Administration->Windows Azure->Add and Edit Subscription wizards. The three profiles are the certificate blob (a double-base64 encoded copy of the certificate), the certificate password (used to decrypt the private key), and the credentials to use with the proxy for the subscription. Or by their element IDs:

Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Blob
Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Password
Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Proxy

It took a little reverse engineering to figure how to leverage these in a PowerShell workflow within SCOM, mainly due to trying to figure out the format of the certificate. Here’s the PowerShell from my SCOM discovery that I was able to come up with during testing:

# define certificate blob from run as profile
$CertBlob='$RunAs[Name="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Blob"]/Data$'
$CertPassword='$RunAs[Name="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Password"]/Password$'
$AzureSubId='$Target/Property[Type="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.Subscription"]/SubscriptionId$'
$AzureSubName='$Target/Property[Type="System!System.Entity"]/DisplayName$'

# decode certificate blob from run as profile
$tempBytes=[System.Convert]::FromBase64String($CertBlob) # once
$tempString=[System.Text.Encoding]::Unicode.GetString($tempBytes)
$tempBytes=[System.Convert]::FromBase64String($tempString) # twice
$AzureCert=New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $AzureCert.Import($tempBytes,$CertPassword,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::MachineKeySet)

* Sorry for the word-wrap. I’ve done my best to ensure these copy out correctly.

And now there is a x509 certificate object $AzureCert that we can provide to Set-Azuresubscription:

Set-AzureSubscription -SubscriptionId $AzureSubId -Certificate $AzureCert -SubscriptionName $AzureSubId
Select-AzureSubscription -Current $AzureSubId

As for the proxy, Azure PowerShell uses the user or system’s proxy settings. To determine the proxy settings and user context the Windows Azure MP stores the proxy address as a property of the Azure Subscription and the Run As Profile associates to a Run As account that the data source will run as and authenticate (via negotiate presumably) to the proxy as. If no Run As Account is associated, the Default Action Account will be used. If the proxy doesn’t require authentication, then the below example will still work. Make sure to include this in any DataSourceType (WindowsAzure! reference assuming):

RunAs="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.RunAsProfile.Proxy"

Then to use the proxy settings in PowerShell, thanks to several examples found on the Internet ultimately culminating to:

$webproxy = "$Target/Property[Type="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.Subscription"]/ProxyServerAddress$"
if ($webproxy -notmatch "^https://") { $webproxy = "https://" + $webproxy }
$proxy = new-object System.Net.WebProxy
$proxy.Address = $webproxy
$proxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials
[System.Net.WebRequest]::DefaultWebProxy = $proxy

And believe it or not Azure requests will be sent via the proxy. Next, we need some discovery data. I created a class name Azure.WebApp, and here’s the discovery bag script:

[string]$SourceId = "$Config/SourceId$"
[string]$ManagedEntityId="$Config/ManagedEntityId$"
$api = New-Object -ComObject ‘MOM.ScriptAPI’
$discoveryData = $api.CreateDiscoveryData(0, $SourceId, $ManagedEntityId)

foreach ($Website in Get-AzureWebsite) {
    $Instance = $discoveryData.CreateClassInstance("$MPElement[Name='Azure.WebApp']$")
    $Instance.AddProperty("$MPElement[Name='System!System.Entity']/DisplayName$", $Website.Name)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/Enabled$", $Website.Enabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/State$", $Website.State.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/UsageState$", $Website.UsageState.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/HostNames$", $Website.HostNames -join ';')
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/EnabledHostNames$", $Website.EnabledHostNames -join ';')
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/AdminEnabled$", $Website.AdminEnabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/Sku$", $Website.Sku.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/RepositorySiteName$", $Website.RepositorySiteName)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/RepositoryUri$", $(($Website.SiteProperties.Properties | ? {$_.Name -eq "RepositoryUri"}).Value))
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/WebSpace$", $Website.WebSpace)
    $discoveryData.AddInstance($Instance)
}

$api.Return($discoveryData)
#$discoveryData

Note the return—$api.Return($discoveryData) instead of the commented #$discoveryData. I use that because instead of using the “native” Microsoft.Windows.PowerShellDiscoveryProbe, I call powershell.exe with System.CommandExecuterProbePropertyBagBase instead (you’ll see several of the newer MPs doing this too). The reason to call out to powershell.exe instead of using the native module is that native module is using a PowerShell Workspace locked in at host version 2.0 and Azure PowerShell requires 3.0.

Altogether my current class type and data source type for the discovery looks like (note the hosting relationship with the subscription class):

      https://www.w3.org/2001/XMLSchema" />
      https://www.w3.org/2001/XMLSchema" />
      https://www.w3.org/2001/XMLSchema" />
      https://www.w3.org/2001/XMLSchema" />
      https://www.w3.org/2001/XMLSchema" />
      https://www.w3.org/2001/XMLSchema" />











            $Config/IntervalSeconds$
            %windir%\system32\WindowsPowerShell\v1.0\powershell.exe

            -NoLogo -NoProfile -Noninteractive ". '$File/DiscoverAzureWebApps.ps1$'"
            $Config/TimeoutSeconds$
            true


                DiscoverAzureWebApps.ps1
                param(
    [string]$SourceId = "$Config/SourceId$"
    ,[string]$ManagedEntityId="$Config/ManagedEntityId$"
    ,[string]$debug="$Config/Debug$"
)

# normalize debug to bool
if ($debug -eq "1" -or $debug.ToLower() -eq "true") {
$debug = $true
} else {
$debug = $false
}

# configure proxy
$webproxy = "$Target/Property[Type="WindowsAzure!Microsoft.SystemCenter.WindowsAzure.Subscription"]/ProxyServerAddress$"

If ($webproxy -ne "") {
    if ($webproxy -notmatch "^https://") { $webproxy = "https://" + $webproxy }
    if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 701, 0, "Using proxy: $webproxy") }
    $error.Clear()
    $proxy = new-object System.Net.WebProxy
    $proxy.Address = $webproxy
    #$account = new-object System.Net.NetworkCredential($user,[Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pwd)), "")
    #$proxy.credentials = $account
    $proxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials
    [System.Net.WebRequest]::DefaultWebProxy = $proxy
    if ($error.Count -gt 0) {
        if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 703, 0, "Error configuring proxy: $error") }
    }
}

# api and discovery bag staging
$api = New-Object -ComObject 'MOM.ScriptAPI'
$DiscoveryData = $api.CreateDiscoveryData(0, $SourceId, $ManagedEntityId)

# import module for Azure (hardcoded for now)
$error.Clear()
import-module '$Config/PathToAzurePSModule$'
if ($error.Count -gt 0) {
if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 704, 0, "Error importing Azure PowerShell module: $error") }
exit 1
}

# decode certificate blob from run as profile
$tempBytes=[System.Convert]::FromBase64String($CertBlob)
$tempString=[System.Text.Encoding]::Unicode.GetString($tempBytes)
$tempBytes=[System.Convert]::FromBase64String($tempString)
$AzureCert=New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$AzureCert.Import($tempBytes,$CertPassword,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::MachineKeySet)
if ($error.Count -gt 0) {
if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 704, 0, "Error reading Azure certificate: $error") }
exit 1
}

# add Azure subscription, name it after the ID to keep it simple and unique
$error.Clear()
Set-AzureSubscription -SubscriptionId $AzureSubId -Certificate $AzureCert -SubscriptionName $AzureSubId
if ($error.Count -gt 0) {
if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 704, 0, "Error setting Azure subscription: $error") }
exit 1
}

# select Azure subscripton as current
Select-AzureSubscription -Current $AzureSubId

$error.Clear()
# loop through web apps and create a discovered instance for each
foreach ($WebApp in Get-AzureWebsite) {
    $Instance = $DiscoveryData.CreateClassInstance("$MPElement[Name='Azure.WebApp']$")
    $Instance.AddProperty("$MPElement[Name='System!System.Entity']/DisplayName$", $WebApp.Name)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/Enabled$", $WebApp.Enabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/State$", $WebApp.State.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/UsageState$", $WebApp.UsageState.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/HostNames$", $WebApp.HostNames -join ';')
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/EnabledHostNames$", $WebApp.EnabledHostNames -join ';')
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/AdminEnabled$", $WebApp.AdminEnabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/Sku$", $WebApp.Sku.ToString())
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/RepositorySiteName$", $WebApp.RepositorySiteName)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/RepositoryUri$", $(($WebApp.SiteProperties.Properties | ? {$_.Name -eq "RepositoryUri"}).Value))
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/WebSpace$", $WebApp.WebSpace)
    $Instance.AddProperty("$MPElement[Name='WindowsAzure!Microsoft.SystemCenter.WindowsAzure.Subscription']/SubscriptionId$", $AzureSubId)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/NumberOfWorkers$", $WebApp.NumberOfWorkers)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/WebSocketsEnabled$", $WebApp.WebSocketsEnabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/Use32BitWorkerProcess$", $WebApp.Use32BitWorkerProcess)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/DefaultDocuments$", $WebApp.DefaultDocuments -join ';')
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/NetFrameworkVersion$", $WebApp.NetFrameworkVersion)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/PhpVersion$", $WebApp.PhpVersion)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/RequestTracingEnabled$", $WebApp.RequestTracingEnabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/HttpLoggingEnabled$", $WebApp.HttpLoggingEnabled)
    $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/DetailedErrorLoggingEnabled$", $WebApp.DetailedErrorLoggingEnabled)
    if ($WebApp.HostNames.Count -gt 1) {
        $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/URL$", "https://$($WebApp.HostNames[0])")
    } else {
        $Instance.AddProperty("$MPElement[Name='Azure.WebApp']/URL$", "https://$($WebApp.HostNames)")
    }
    $DiscoveryData.AddInstance($Instance)
    if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 700, 0, "Added Azure WebApp Instance: $($WebApp.Name)") }
}
if ($error.Count -gt 0) {
    if ($debug) { $api.LogScriptEvent("DiscoverAzureWebApps.ps1", 704, 0, "Error reading Azure Web Apps: $error") }
    exit 1
}
$api.Return($DiscoveryData)
]]>









    System!System.Discovery.Data

Well that’s a start and should help lay some ground work for custom authoring of Azure monitoring. Next round I’ll finish this up with some monitors and talk about ideas around distributing and monitoring for the expected version of Azure PowerShell.

Resources

https://blogs.msdn.com/b/walterm/archive/2013/04/13/first-impressions-on-system-center-management-pack-for-windows-azure-fabric-preview-for-scom-2012-sp1.aspx

https://blogs.technet.com/b/heyscriptingguy/archive/2013/06/22/weekend-scripter-getting-started-with-windows-azure-and-powershell.aspx

UNIX/Linux Log File Monitor Alert Description Tips

cchamp — Fri, 09 May 2014 18:54:53 GMT

Hello! I recently had the opportunity of working with a customer who had a pretty simple ask about log file monitoring. When using the UNIX/Linux Log File Monitor Management Pack template, how do we include more than just one line that matched, and include all lines that matched since the last interval?

This is a really good question, and something you might expect out of the box. There is however some unfavorable behavior with the underlying data source when expecting more than one matching result. In this post I’ll discuss what causes the default behavior of only including one line, and how we can expand that to include additional data that may have matched our log file monitor (er rule).

First a few assumptions. This addresses rules that have been created using the UNIX/Linux Log File Monitoring management pack template in System Center 2012 R2 Operations Manager, and the configuration in the rule the template creates has not been modified. For reference the default configuration will look something like:

$Target/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$
/var/log/messages
$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$
$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$
error 1337
false

And the default alert description will look like:

$Data/EventDescription$

To find those settings, select the Authoring->Management Pack Objects->Rules view, and scope it to UNIX/Linux Computer in the UNIX/Linux Core Library. The LogFile Template rules should show here and be disabled by default (overrides exist to enable to the computer or group specified in the template configuration). Look at the properties of the rule, and under the Configuration tab there are two sections to note. The first contains the Data Source, named something similar to “Log File VarPriv Datasource”. This configures the settings for the UNIX/Linux log file module. The second is the Response, which will contain one named GenerateAlert. This configures the alert settings just as any other console-authored rule or monitor.

I mentioned before about some possible unfavorable behavior with the data source, but what does that necessarily mean? In the case of the ask, the alert description contains only one matching line. That’s because the data source feeds the output of the log module to an event mapper. I assume that is done to bring uniformity to the way Windows events and UNIX logs could potentially be stored and reported against, but by using the log file monitoring template, there are no responses for saving the data in the OperationsManager DB or DW as event data present by default. Below is the data source for reference, this was taken from the UNIX/Linux Core Library MP version 7.5.1019.0. I’ve highlighted the cause of the description only containing one line.

    https://www.w3.org/2001/XMLSchema" />
    https://www.w3.org/2001/XMLSchema" />
    https://www.w3.org/2001/XMLSchema" />
    https://www.w3.org/2001/XMLSchema" />
    https://www.w3.org/2001/XMLSchema" />
    https://www.w3.org/2001/XMLSchema" />









          https
          $Config/Host$
          $Config/UserName$
          $Config/Password$
          $Config/LogFile$
          $Config/RegExpFilter$
          $Config/IndividualAlerts$
          $Target/ManagementGroup/Name$
          300
          false
          false


          $Target/Id$
          $MPElement$
          WSManEventProvider
          WSManEventProvider

          0
          3
          0

          Detected Entry: $Data///row$










System!System.BaseData

To change this behavior though, we want to focus on the GenerateAlert response. Click Edit to view the settings, and locate the Alert description field. There are two ways we might accomplish our goal. The first is we can include ALL lines that have matched in the past interval, but all the lines appear run together on a single line. Or we can include up to the first 10 matching lines by addressing those first 10 rows of the data returned individually.

To get all the data collapsed on a single line, use the following as the description:

Detected Entries: $Data///SCXLogProviderDataSourceData$

To get only the first 10 matches, but each on their own line, use this as the description:

Up to the first 10 matching entries:
$Data///row[1]$
$Data///row[2]$
$Data///row[3]$
$Data///row[4]$
$Data///row[5]$
$Data///row[6]$
$Data///row[7]$
$Data///row[8]$
$Data///row[9]$
$Data///row[10]$

For reference, here is what the Alert Context for one of these rules looks like. This is the data that will be processed for the variable substitution in the alert description.

< DataItem type =" SCXLogProviderDataSourceData " time =" 2014-05-09T20:52:59.9407442-04:00 " sourceHealthServiceId =" 146F6C57-9775-06DE-9590-B2B9A2D11920 " >
< SCXLogProviderDataSourceData >
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 1.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 2.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 3.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 4.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 5.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 6.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 7.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 8.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 9.
< row > May 9 20:50:30 centos6 root: error 1337: ltnz overload, 2-1337-2 proceed to level 10.

Looking at the above, and if you have any experience with XPath queries, you can see the alert description settings proposed above are nothing too complicated. The first method above addresses the SCXLogProviderDataSourceData XML node, and returns everything below on one line. The second method calls out to the row node and calls out to the [1], [2], [3], … instance of the row entry. If only 3 rows are returned, the other 7 will remain blank and thankfully not result in an error.

Follow up question. Now with the matching lines in the alert, how do we generate a new alert for every set of matches found, but not for each line found? By default, the Data Source is configured with IndividualAlerts set to false which causes any future log matches by the rule to increment the existing alert’s repeat count. Whereas when IndividualAlerts is set to true, each line creates its own alert. We can address this using the alert suppression settings (view the rule properties/configuration, and modify the GenerateAlert response where there will be an Alert Settings button at the bottom). And we only need one entry:

$Data///SCXLogProviderDataSourceData$

By adding that line above, it forces an additional evaluation to the alert suppression, and assuming there are date/time stamps in the log file, even if the same set of errors match, we’ll receive a new alert. Be sure to close any open alerts that were raised prior to modifying the suppression settings. Otherwise the alerts will still enforce suppression as it was configured at the time the alert was raised.

Using this information you should be able to custom tailor the alert description better than what the template provides out of the box, and what I really appreciate about these methods is that it can all be done from the Operations Console.

Now for the instant gratification types, screenshots!

BEFORE:

AFTER ON A SINGLE LINE:

AFTER WITH ROWS:

Known Issues

Once you modify the alert settings, you should not edit the rule using the Management Pack Template. In fact, once you have the log file rules dialed-in, I would open the MP XML and remove any line that begins with and tags if you don’t have any legitimate folders for Monitoring views.
IF YOU DO EDIT THE MP TEMPLATE, YOU WILL OVERWRITE CHANGES MADE TO THE RULE AND ALERT DESCRIPTION WITH THE DEFAULTS.
There is a limit to ten lines using the individual row method. The limit is imposed by ten alert parameters max. You could increase this by passing as many rows as you could squeeze into the event data mapper data source by copy/pasting from the UNIX/Linux Core Library (and grab Microsoft.Unix.SCXLog.Native.DataSource too since it will be needed and is marked internal), or even insert a custom data source for post-processing. This is more ideal if you consistently author using more robust authoring tools and/or the raw XML.
Technically we are duplicating what can amount to a significant amount of data depending on how often the alert triggers. The log entries will be stored in the alert context field, as well as the alert parameters field. Generally speaking this shouldn’t be too much of a concern, but it’s worth mentioning in case anyone is planning on alert archives with 10’s of 1000’s of these.

Resources

Adding custom information to alert description (s) and notifications – An oldie, but goodie on Kevin Holman’s blog. My go to link for alert variable reference.

AlertParameters (UnitMonitor) – This is good reference on the fact there are only 10 alert parameters. The AlertSettings block is different for rules, so beware.

SCOM 2012: Authoring UNIX/Linux Log File Monitoring Rules – Good article on custom authoring Log File Monitoring rules, plus details on using the correlated condition detect module with UNIX/Linux logs (need x occurrences in y time).

How to find Subscriptions with deleted Rules and Monitors

cchamp — Mon, 10 Mar 2014 19:35:12 GMT

… Or an introduction to subscription criteria.

What I want to share today is a PowerShell snippet I've been sitting on that checks out the current subscriptions for rules or monitors that no longer exist. With some of the newer management packs replacing their old versions, this can really help save some time in tracking down one-off subscriptions that tie to those old rules and monitors. Also, it's always good to check this before performing any major migration or upgrade say from 2007 R2 to 2012. And there is always the case you might find yourself trying to edit a complex subscription and it’s resulting in a System.NullReferenceException error in the console.

Without further ado, the snippets…

2007 R2:

foreach ($notificationsub in Get-NotificationSubscription) {
    $notificationsubname=$notificationsub.DisplayName
    $writesub = $true
    $criteria = [xml]$notificationsub.Configuration.Criteria.ToString()
    $xmlnsmgr = New-Object System.Xml.XmlNamespaceManager $criteria.CreateNavigator().NameTable
    foreach ($simpexpr in $criteria.SelectNodes("//SimpleExpression", $xmlnsmgr)) {
        $property = ""
        $guid=0
        $element=$null
        $property = $simpexpr.GetElementsByTagName("Property").Item(0)."#text".ToString()
        if ($property -eq "RuleId" -or $property -eq "ProblemId") {
            $value = $simpexpr.GetElementsByTagName("Value").Item(0)."#text".ToString()
            try {
                $guid=[guid]$value
            } catch {}            
        }
        if ($guid -ne 0) {
            if ($property -eq "RuleId") {
                $element=Get-Rule -Criteria "Id = '$guid'"
            } else {
                $element=Get-Monitor -Criteria "Id = '$guid'"
            }
            if ($element -eq $null) {
                if ($writesub) {
                    Write-Output ""
                    Write-Output "Subscription: $notificationsub"
                    Write-Output "Name: $notificationsubname"
                    Write-Output "------------------------------------------------------------------------------"
                    $writesub=$false
                }
                if ($property -eq "RuleId") {
                    Write-Output "Rule: $guid"
                } else {
                    Write-Output "Monitor: $guid"
                }
            }
        }
    }
    if (!$writesub) {
        Write-Output ""
    }
}

2012:

foreach ($notificationsub in Get-SCOMNotificationSubscription) {
  $notificationsubname=$notificationsub.DisplayName
  $writesub = $true
  if ($notificationsub.Configuration.Criteria -ne $null) {
    $criteria = [xml]$notificationsub.Configuration.Criteria.ToString()
    $xmlnsmgr = New-Object System.Xml.XmlNamespaceManager $criteria.CreateNavigator().NameTable
    foreach ($simpexpr in $criteria.SelectNodes("//SimpleExpression", $xmlnsmgr)) {
        $property = ""
        $guid=0
        $element=$null
        $property = $simpexpr.GetElementsByTagName("Property").Item(0)."#text".ToString()
        if ($property -eq "RuleId" -or $property -eq "ProblemId") {
            $value = $simpexpr.GetElementsByTagName("Value").Item(0)."#text".ToString()
            try {
                $guid=[guid]$value
            } catch {}            
        }
        if ($guid -ne 0) {
            if ($property -eq "RuleId") {
                $element=Get-SCOMRule -Id $guid
            } else {
                $element=Get-SCOMMonitor -Id $guid
            }
            if ($element -eq $null) {
                if ($writesub) {
                    Write-Output ""
                    Write-Output "Subscription: $notificationsub"
                    Write-Output "Name: $notificationsubname"
                    Write-Output "------------------------------------------------------------------------------"
                    $writesub=$false
                }
                if ($property -eq "RuleId") {
                    Write-Output "Rule: $guid"
                } else {
                    Write-Output "Monitor: $guid"
                }
            }
        }
    }
    if (!$writesub) {
        Write-Output ""
    }
   }
}

Well that’s great and all, but how do we use it?

In either case of 2007 or 2012, either save the above script to a PS1 file or copy/paste it directly into the Operations Manager Shell.

Once run, one of two things may happen:

1) No output. Great! No problem.

2) Output…

Subscription: Subscription26c89132_b882_4d51_9924_74724e3bed9e
Name: Broken Sub
------------------------------------------------------------------------------
Rule: 36f41859-7246-60c9-1a24-834c9597fc42

Okay, so it found a subscription and returned a Rule’s GUID. It can’t return the name of the rule since it no longer exists. To fix this, either modify the Notification Internal Library MP XML or just remove the entire subscription and rebuild it from scratch.

Note, as tested in the 2012 SP1 console, when criteria contains a missing rule/monitor and it was the only one specified in the criteria, the UI will present it as though no criteria for the specific rule/monitor has been specified. If multiple rules/monitors are specified and one is missing, the console will not allow you to modify the criteria. Although different factors could result in different behavior, and therefore experiences may vary.

In order to modify the MP XML with this rule, first export the Notifications Internal Library MP aka Microsoft.SystemCenter.Notifications.Internal.xml.

Then search for the GUID returned by the PowerShell, in the example this was 36f41859-7246-60c9-1a24-834c9597fc42. It should be found in a block:

              
                
                  "https://www.w3.org/2001/XMLSchema" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
                    
                      RuleId
                    
                    Equal
                    
                      36f41859-7246-60c9-1a24-834c9597fc42

In this case that rule is the only criteria, so to go a step further, the whole rule can be deleted (everything between and including the tags). Then also remove the associated DisplayString block near the end of the MP.

In the case where it is not the only criteria, it can get a bit complex:

              
                
                  "https://www.w3.org/2001/XMLSchema" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
                    
                      
                        
                          
                            
                              ProblemId
                            
                            Equal
                            
                              d064c5b4-1ca9-a551-04b3-8a6d6ca9919f
                            
                          
                        
                        
                          
                            
                              RuleId
                            
                            Equal
                            
                              36f41859-7246-60c9-1a24-834c9597fc42
                            
                          
                        
                      
                    
                    
                      
                        
                          Severity
                        
                        Equal
                        
                          2

In short we need to remove the block that contains the missing rule, but in the above case we have additional criteria that takes some careful consideration. And that is that the Rule/Monitor criteria is wrapped in an block, with additional criteria contained in an block. A quick note about and blocks, they require at least two blocks. With that in mind, if block containing the missing rule is removed, the MP XML will become invalid. In that case we need to remove the and tags, along with their matching closing tags, but leaving the part of the desired criteria (the ProblemId expression). Since that barely made sense typing it, this is what the end result would look like:

              
                
                  "https://www.w3.org/2001/XMLSchema" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
                        
                          
                            
                              ProblemId
                            
                            Equal
                            
                              d064c5b4-1ca9-a551-04b3-8a6d6ca9919f
                            
                          
                        
                    
                      
                        
                          Severity
                        
                        Equal
                        
                          2

Be cautious when there is only one rule/monitor specified, but other criteria is present such as severity. If the rule/monitor criteria is removed, but the severity match remains, then it would change the subscription to match all alerts raised with that particular severity.

Once the XML has been modified to fit the new desired criteria, try to import the MP (preferably in a Lab/Dev/QA environment). If an error is returned hopefully it is descriptive enough to fix the issue. If not retrace the steps to make sure all XML tags have a valid start/end, and that there are no orphaned DisplayStrings if a Rule was removed. Once it imports, the issue should resolved.

What if:

- I’m not too good modifying XML: I recommend deleting and rebuilding the subscription.

- I have hundreds of these!

Theoretically this could be automated, but it would take a pretty fair understanding of XML and the criteria schema to check for the aforementioned scenarios, as well as any others I didn’t consider. The nice thing is if the criteria is broken, it won’t import the management pack. If there’s interest in an automated 2012 version, feel free to leave a comment and let me know.

Here are some resources that may be of some use:

ExpressionType Reference
https://msdn.microsoft.com/en-us/library/jj130463.aspx

Configuring Operations Manager 2007 R2 Product Connector Subscription Advanced Criteria – In case you came across this article to gain a deeper understanding notification subscription criteria or the ExpressionType syntax
https://support.microsoft.com/?kbid=2026093

“remove section of an XML file” – Where I might start if I were to automate this process. Keep in mind portions of criteria in And/Or statements that are intended to remain will need saved to a new node before the entire And/Or nodes are deleted.
https://social.technet.microsoft.com/Forums/windowsserver/en-US/702f3e4a-df64-4a97-af8b-1e247b6633e2/remove-section-of-an-xml-file?forum=winserverpowershell

Last, but not least: This is information is provided as-is and provides no warranty. Always maintain a backup of your management packs, and never work on notification subscriptions in a production environment.

Example MP to Trigger Non-stop Notifications

cchamp — Wed, 04 Nov 2009 12:48:00 GMT

Occasionally I speak to someone who would like to have OpsMgr continually send notifications until an alert is resolved. One way this can be done is by setting up multiple notification subscriptions and staggering the alert aging. The problem with this method is it is finite, and can be quite tedious to setup.

Alternately, we can have a process run one a timed interval to modify certain open alerts in a manner that would trigger another notification to be sent. One way to trigger that is to update a custom field.

So as an example of how to accomplish this, here is a small management pack that includes a Timed Command Rule that executes a script (VBS with Powershell embedded) on the RMS. The script is setup to run every 5 minutes, and will update Custom Field 10 in order to trigger a new notification when it is run. If anything exists in Custom Field 10 already it will be overwritten. The MP should work with OpsMgr 2007 SP1 and newer.

Please note that the management pack in its current form can be detrimental to performance. It will affect ALL alerts that are not closed (resolution state 255). If you have quite a few open alerts, it could potentially consume quite a bit of CPU to update the alerts, and also consume additional resources when the notification workflow runs to send out the notifications. Not to mention any adverse effects that could be caused by the increased size of the alert history which will grow with each run.

This is of course provided AS-IS and as an EXAMPLE only.

Custom.Field.Updater.xml