Accidental Denial of Service through Inefficient Program Design Part 1 – Watching WMI Events for Object Creation (e.g. Win32_Process)

  

There are few things that are more annoying as a user than to have the performance of a computer which they’re using grind to a halt.  This series will outline program design flaws that I’ve run across, explain what they are, how they impact the system, example scenarios where the impact will be exacerbated, and safer alternatives to accomplish the same programming goal.  In part one of this series, we’ll be exploring:

Watching WMI Events for Object Creation (e.g. Win32_Process)

 

What It Is

WMI stands for Windows Management Infrastructure(2012+) or Windows Management Instrumentation(2008R2 and before).  It is a management infrastructure that allows for local and remote management data collection and operations execution.  It’s a great framework to have around to be used when you need to find out some information and want to be able to do so without writing much if any code.  If you’re at a Powershell prompt or still writing vbscripts like its the 90s, it’s a great option for many one-off tasks.

How It Impacts the System

The framework itself has fairly high overhead compared to native function calls which is required to be able to support the ability to make specific queries.  Since it’s a generic framework, the variance in the amount of resources it requires to do something depends predominantly on the Provider and Class, so one needs to treat its class as its own impact entity when considering its implications on the system.  In other words, the query “Select * From Win32_Product” may take a considerably different amount of resources and time to execute compared to the query “Select * From Win32_Registry”.  When performing an event query, the amount of resources can be multiplied substantially due to the way that some event providers work.

Since the WMI framework is so vast, we’ll be focusing the rest of part one on the Win32_Process class.  This is a commonly used class for performing event queries most often to be notified of process creation.  For example, here’s some C# code that I’ve seen used before to be notified of when a Notepad process has been created:

 var scope = new ManagementScope( string.Format("\\\\{0}\\root\\cimv2", machineName) , null); 
 scope.Connect(); 
 var wmiStartQuery = new WqlEventQuery("__InstanceCreationEvent", 
 new TimeSpan(0, 0, 1), 
 "TargetInstance ISA \"Win32_Process\" AND " + 
 "TargetInstance.Name=\"Notepad.exe\" "); 
 var StartWatcher = new ManagementEventWatcher(scope, wmiStartQuery); 
 StartWatcher.EventArrived += StartWatcher_EventArrived; 
 StartWatcher.Start();

Let’s take a step back here and look at what is actually happening.  The root\cimv2 namespace is being connected to on the machine “machineName.”  WqlEventQuery is being instantiated to report events of class “__InstanceCreationEvent”, send status within one second, and constrain events reported to come from the class Win32_Process and have a Name property equal to “Notepad.exe.”  The ManagementEventWatcher uses the WMI connection on the machine to perform that query and calls the EventArrived event whenever data is available.  Looking at the explanation for the second parameter in the documentation linked above, we see:

This value is used in cases where there is no explicit event provider for the query requested and WMI is required to poll for the condition. This interval is the maximum amount of time that can pass before notification of an event must be delivered.

So let’s translate this into what basically actually happens on “machinename” in order to facilitate this translated into C# (accurate enough to show impact of what’s generally happening anyway since obviously WMI doesn’t use WMI to obtain process information:-)):

 Stopwatch sw = new Stopwatch(); 
 var scope = new ManagementScope( 
 string.Format("\\\\{0}\\root\\cimv2", machineName) , null); 
 scope.Connect(); 
 var processQuery = new SelectQuery( "SELECT * FROM Win32_Process WHERE " + 
 "Name=\"Notepad.exe\" "); 
 var searcher = new ManagementObjectSearcher(scope, processQuery); 
 ManagementObjectCollection baseCollection; 
 ManagementObjectCollection nextCollection; 
 sw.Restart(); 
 baseCollection = searcher.Get(); 
 do { 
 TimeSpan timeLeft = TimeSpan.FromSeconds(1).Subtract(sw.Elapsed); 
 if(timeLeft.TotalMilliseconds > 0) Thread.Sleep(timeLeft); 
 nextCollection = searcher.Get(); 
 sw.Restart(); 
 foreach(ManagementObject moNext in nextCollection) { 
 bool matched = false; 
 uint procId = (uint)moNext["ProcessId"]; 
 foreach(ManagementObject moBase in baseCollection) { 
 if((uint)moBase["ProcessId"] == procId) { 
 matched = true; break; 
 } 
 } 
 if(!matched) { 
 //Report that the process was created here! 
 } 
 } 
 baseCollection = nextCollection; 
 } 
 while (isRunning);

It’s pretty easy to see how that is inefficient.  Sure, if you write the C# code shown originally, your callback will only fire when the criteria is met, but on the machine where you are monitoring in the WMI Service Host, an inefficient operation is taking place on a continual basis.  If you use a shorter timeout, this querying has to take place even more frequently.  If you use a longer timeout, there is potentially longer delay between when the process is created and when you are notified of its creation.

Example Scenario of Exacerbation

Consider the following scenario:

  • Application A needs to be launched whenever Application B is running
  • Application B could potentially be ran on a server with Remote Desktop Services by users in specific sessions
  • Application C is launched in each session when the session is started so that it can launch Application A in the proper context in that session whenever Application B is launched
  • Multiple users log on to the same server with a Remote Desktop Services session

In this case, multiple event queries could end up running simultaneously.  Even if they are the same query text, WMI isn’t going to combine them into a single query with multiple subscribers since each user will be connected to WMI with a different RPC context.

On a system with 8 vCPUs, 32 GB of RAM, and SSD storage for the OS Drive and Pagefile, I observed the following by measuring the impact of having X users perform the example event query with 133 base processes running and 7 processes added per signed in user (I was only adding one process per user, but there are dependencies for running a session as well):

Users Performing Event Query Timeout (seconds) % CPU Used by WMI as shown in Task Manager over Two Minutes
2 1 5.8% – 7.0%
6 1 13.7%-14.5%
15 1 49.6%-50.8%

As you can see, as more and more users get on, more and more CPU is being eaten up just to maintain this event query, and application response times will suffer more and more.   Eventually, with enough users, it would become a nightmare to interact with an application on the machine since there would be so many context switches and contention for CPU time.

Safer Alternatives

In this case, while it’s still far from an optimal programming practice, increasing the timeout value can massively mitigate the impact.  Running the same test as above except changing the timeout value to 30 seconds, with 15 users the average % CPU impact was shown to be 0% and less than 5 seconds of total CPU time was used by WMI in two minutes (with 8 cores that means it used between .25% and .31% of available CPU in that time period).  If an application doesn’t need to know right away, a longer timeout is something easy to change to make this methodology much more palatable.  If a process doesn’t live very long, the chances of it failing to be caught does go up the longer the polling interval is.

Either in conjunction with the above or on its own, performing a single event query from a service and using that service to launch applications in the correct context is also a huge improvement in resource utilization in a Remote Desktop Services scenario.

When running on a local machine, using WMI is just adding overhead from a resource standpoint versus calling the native APIs yourself, so switching the logic to native code will also provide some benefit.  Using the Process Status API, Tool Help Library, or NtQuerySystemInformation with the SystemProcessInformation parameter will get a user mode application a collection of every process running on the local machine for supported versions of the Windows Server family of operating systems at the time of this point.

Ideally, polling would be avoided altogether and there is of course a mechanism to do that as well.  One can use a driver to replace Application A in the scenario above and use either PsSetCreateProcessNotifyRoutine or PsSetCreateProcessNotifyRoutineEx to register a function for Windows to call each time a new process is created (a la procmon and sysmon).

Enabling the Audit Process Creation policy will cause an event to be logged to the security event log whenever a process starts.  Once that is happening, you can get notified of when said event log is written to using EvtSubscribe or even the managed class System.Diagnostics.Eventing.Reader.EventLogWatcher.

System Monitor also has an event that will get trigged when a counter is added to a collection, so monitoring the Process counters will allow you to get notified that a process has been created.  This does add the overhead of COM and a UI as well, so its impact can add up on Remote Desktop Services too.

Perhaps the best thing to do for this particular scenario would be to eschew a programmatic process watching solution altogether.  If you know that you need an application launched alongside another, it makes sense to use a shortcut to a batch file, script file, or executable file to launch both applications.

Follow us on Twitter, www.twitter.com/WindowsSDK.