PowerShell SCOM monitor using too much resources

Question

Hi all, I hope you could help me.

I have built custom IIS Web Site and IIS App Pool availability monitors to replace the original ones. The reason is that the original do not have an option to alert only after certain period of object being down, and we have a lot of IIS restarts that cause alerts.

I have used Kevin Holman's fragments in Visual studio and the whole thing have two, pretty much the same, PowerShell script based monitors. One for App Pools and one for Web Sites. At the beginning I haven't build with cookdown and I though that was the reason. Once implemented, the cookdown has reduced resource usage a lot, but I still get a lot of strain on agents.
For example, all my agents on IIS web servers are restarting all the time, and Monitoring host private bytes are looking like this:

If I disable one of these monitors, I will get slightly less impact on performance, but only after disabling both, it will come back to normal.

The scripts are very simple. I am including only logic part, not the whole script.

App Pool Availability Monitor

# Begin MAIN script section  
#=================================================================================  
  
$strCondition = Get-WebAppPoolState  
  
foreach ($strCond in $strCondition) {  
  
    $bag = $momapi.CreatePropertyBag()  
    $PoolID = $strCond.ItemXPath.split("'")[1]  
    $AppPoolStatus = $strCond.Value  
    $bag.AddValue('PoolID',$PoolID)  
  
    #Check the value of $strCond  
    IF ($AppPoolStatus -eq 'Started') {  
    $momapi.LogScriptEvent($ScriptName,$EventID,0,'Good Condition Found')  
    $bag.AddValue('Result','Running')  
    } else {  
    $momapi.LogScriptEvent($ScriptName,$EventID,0,'Bad Condition Found')  
    $bag.AddValue('Result','NotRunning')  
    }  
  
    $bag  
}  
  
# End of script section

Web Site Availability Monitor

# Begin MAIN script section  
#=================================================================================  
  
$strCondition = Get-WebSiteState  
  
foreach ($strCond in $strCondition) {  
  
    $bag = $momapi.CreatePropertyBag()  
    $DisplayName = $strCond.ItemXPath.split("'")[1]  
    $WebSiteStatus = $strCond.Value  
    $bag.AddValue('DisplayName',$DisplayName)  
  
    #Check the value of $strCond  
    IF ($WebSiteStatus -eq 'Started') {  
    $momapi.LogScriptEvent($ScriptName,$EventID,0,'Good Condition Found')  
    $bag.AddValue('Result','Running')  
    } else {  
    $momapi.LogScriptEvent($ScriptName,$EventID,0,'Bad Condition Found')  
    $bag.AddValue('Result','NotRunning')  
    }  
  
    $bag  
}  
  
# End of script section

I am wondering why is this using so much resources and is there a way for me to pinpoint where it happens?

I have also tried to override an agent and add bigger values to Private Bytes and Handle Count, but it just kept on rising until agent restarted.

Also, if there are alternatives that solve initial problem, an other way to get Web Site and App Pool availability monitor that you can control in terms of alert suppression.

Thanks!
Niksa

Accepted Answer

Okay, I have found a solution, or rather a workaround.

Instead of using IIS Webadministration PS module cmdlets, I have found alternative ways to get IIS App Pool and IIS Web Site status data.

This is how my PowerShell part looks like in each:

App Pool Availability Monitor

# Begin MAIN script section
#=================================================================================


$counters = Get-Counter -Counter "\APP_POOL_WAS(*)\Current Application Pool State"
$strCondition = $counters.countersamples | where InstanceName -ne "_total"

foreach ($strCond in $strCondition) {

    $bag = $momapi.CreatePropertyBag()
    $PoolID = $strCond.InstanceName
    $AppPoolStatus = $strCond.RawValue
    $bag.AddValue('PoolID',$PoolID)

    #Check the value of $strCond
    IF ($AppPoolStatus -eq '3') {
        $momapi.LogScriptEvent($ScriptName,$EventID,0,'Good Condition Found')
        $bag.AddValue('Result','Running')
    } else {
        $momapi.LogScriptEvent($ScriptName,$EventID,0,'Bad Condition Found')
        $bag.AddValue('Result','NotRunning')
    }

    $bag
}                                   


# End of script section

Web Site Availability Monitor

# Begin MAIN script section
#=================================================================================


$strCondition = Invoke-Expression "$env:SystemRoot\system32\inetsrv\AppCmd.exe list site"

foreach ($strCond in $strCondition) {

    $bag = $momapi.CreatePropertyBag()
    $DisplayName = $strCond.Split('"')[1]
    $WebSiteStatus = $strCond.Split(':')[-1]
    $bag.AddValue('DisplayName',$DisplayName)

    #Check the value of $strCond
    IF ($WebSiteStatus -eq 'Started)') {     # Started) is okay because that's the result of splitting the string
        $momapi.LogScriptEvent($ScriptName,$EventID,0,'Good Condition Found')
        $bag.AddValue('Result','Running')
    } else {
        $momapi.LogScriptEvent($ScriptName,$EventID,0,'Bad Condition Found')
        $bag.AddValue('Result','NotRunning')
    }

    $bag
}


# End of script section

I still don't understand why those two cmdlets are causing so much overhead, hopefully I will find out one day.

Answer

You don't show how you handle the " after certain period of object being down" part of your monitoring... Just a wild guess, but that could be the cause of your issue if there is some kind of never-exiting loop in your code.

Answer

@CyrAz ,

It's in the fragment. Below is the whole "IIS Web Site Availability" fragment from management pack. See "MatchCount" part.

I should add that the monitors are working as intended.

  
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
 $Config/IntervalSeconds$  
 $Config/SyncTime$  
   
   
   
   
   
 CORP.IIS8.IIS8WebSiteAvailabilityCustom.Monitor.DataSource.ps1  
   
 #=================================================================================  
 #  Describe Script Here  
 #  
 #  Author:  
 #  v1.0  
 #=================================================================================  
  
  
 # Constants section - modify stuff here:  
 #=================================================================================  
 # Assign script name variable for use in event logging.  
 $ScriptName = "CORP.IIS8.IIS8WebSiteAvailabilityCustom.Monitor.DataSource.ps1"  
 $EventID = "1234"  
 #=================================================================================  
  
  
 # Starting Script section - All scripts get this  
 #=================================================================================  
 # Gather the start time of the script  
 $StartTime = Get-Date  
 #Set variable to be used in logging events  
 $whoami = whoami  
 # Load MOMScript API  
 $momapi = New-Object -comObject MOM.ScriptAPI  
 # Load PropertyBag function  
 # $bag = $momapi.CreatePropertyBag()  WE ARE CREATING PROPERTY BAG FOR EACH OBJECT BELOW  
 #Log script event that we are starting task  
 $momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Script is starting. `n Running as ($whoami).")  
 #=================================================================================  
  
  
 # Begin MAIN script section  
 #=================================================================================  
  
 $strCondition = Get-WebSiteState  
  
 foreach ($strCond in $strCondition) {  
  
 $bag = $momapi.CreatePropertyBag()  
 $DisplayName = $strCond.ItemXPath.split("'")[1]  
 $WebSiteStatus = $strCond.Value  
 $bag.AddValue('DisplayName',$DisplayName)  
  
 #Check the value of $strCond  
 IF ($WebSiteStatus -eq 'Started') {  
 $momapi.LogScriptEvent($ScriptName,$EventID,0,'Good Condition Found')  
 $bag.AddValue('Result','Running')  
 } else {  
 $momapi.LogScriptEvent($ScriptName,$EventID,0,'Bad Condition Found')  
 $bag.AddValue('Result','NotRunning')  
 }  
  
 $bag  
 }  
  
 # End of script section  
 #=================================================================================  
 #Log an event for script ending and total execution time.  
 $EndTime = Get-Date  
 $ScriptTime = ($EndTime - $StartTime).TotalSeconds  
 $momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Script Completed. `n Script Runtime: ($ScriptTime) seconds.")  
 #=================================================================================  
 # End of script  
   
 $Config/TimeoutSeconds$  
   
   
   
   
   
   
   
   
   
 System!System.PropertyBagData  
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
 $Config/IntervalSeconds$  
 $Config/SyncTime$  
 $Config/TimeoutSeconds$  
   
   
   
   
   
   
 Property[@Name="DisplayName"]  
   
 Equal  
   
 $Config/DisplayName$  
   
   
   
   
   
   
   
   
   
 Property[@Name="Result"]  
   
 Equal  
   
 Running  
   
   
   
   
   
   
   
   
 Property[@Name="Result"]  
   
 Equal  
   
 NotRunning  
   
   
   
   
 $Config/MatchCount$  
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
 AvailabilityHealth  
   
 Error  
   
 true  
 Normal  
 MatchMonitorHealth  
   
   
   
   
   
   
   
   
   
 120  
   
 120  
 3  
   
 $Target/Property[Type="System!System.Entity"]/DisplayName$  
   
   
   
   
   
   
   
   
   
   
   
   
   
 Web Site Availability - IIS8 Custom Monitor  
   
   
   
 Good Condition  
   
   
 Bad Condition  
   
   
 Web Site Availability - IIS8 Custom Monitor: Failure  
 Web Site Availability - IIS8 Custom Monitor: detected a bad condition

Answer

I can't say I see anything extroardinarily wrong here, but I would try the following adjustments :

Set a synctime instead of leaving it blank to ensure cookdown is indeed properly enforced
Set a shorter Timeout duration than the running interval

If the issue remains the same, I guess you'll have to take a memory dump of monitoringhost.exe and try to find what objects are staying in memory...

Answer

@CyrAz

I have removed SyncTime completely and have set shorter Timeout than Interval, but no change.
I did process dump, but I have no idea how to debug that, and nothing comes out obvious straight away.

But I managed to pinpoint that the problem is in PowerShell bit, in particular the two pieces of code I shared first.
When I comment out those pieces, Private Bytes come back to normal behavior. Further more, I have narrowed down the problem to lines where $strCondition variables are declared:

For App Pool Availability Monitor:

$strCondition = Get-WebAppPoolState

For Web Site Availability Monitor:

$strCondition = Get-WebSiteState

Why do these two commands leak so much? I have tried to use Remove-Module WebAdministration at the end of the script to unload the module, but it didn't help.

One more thing... On every interval, private bytes of MonitoringHost increase for around 70MB.

PowerShell SCOM monitor using too much resources

5 additional answers