Analyzing a Stuxnet Infection with the Sysinternals Tools, Part 2

In Part 1 I began my investigation of an example infection of the infamous Stuxnet worm with the Sysinternals tools. I used Process Explorer, Autoruns and VMMap for a post-infection survey of the system. Autoruns quickly revealed the heart of Stuxnet, two device drivers named Mrxcls.sys and Mrxnet.sys, and it turned out that disabling those drivers and rebooting is all that’s necessary to disable Stuxnet (barring a reinfection). With Process Explorer and VMMap we saw that Stuxnet injected code into various system processes and created processes running system executables to serve as additional hosts for its payload. By the end of the post I had gotten as far as I could with a snapshot-based view of the infection, however. In this post I continue the investigation by analyzing the Process Monitor log I captured during the infection to gain deeper insight into Stuxnet’s impact on an infected system and how it operates (incidentally, if you like these blog posts, cybersecurity, and books by Tom Clancy and Michael Crichton, be sure to check out my new cyberthriller, Zero Day).

Filtering to Find Relevant Events

Process Monitor captured around 30,000 events while monitoring the infection, which is an overwhelming number of events to individually inspect for clues. Most of the trace actually consists of background Windows activity and operations related to Explorer navigating to a new folder and are not directly related to the infection. Because by default Process Monitor excludes advanced events (paging file, internals IRP functions, System process and NTFS metadata operations), as the status bar indicates, Process Monitor is still showing over 10,000:

The key to using Process Monitor effectively when you don’t know what exactly you’re looking for is to narrow the amount of data to something manageable. Filters are a powerful way to do that and Process Monitor has a filter tailor made for these kinds of scenarios: a filter that excludes all events except ones that modify files or registry keys. You can configure this filter, “Category is Write then Include,” using the Filter dialog:

Events generated by the System process are typically not relevant in troubleshooting cases, but I know that Stuxnet has kernel-mode components, so to be thorough I had to include events executed in the context of the System process, which is the process in which some device drivers execute system threads. You can remove the default filters by checking the Enable Advanced Output option on the filter menu, but I didn’t want to remove the other default filters that omit pagefile and NTFS metadata operations, so I removed just the System exclusion filter (the second one in the above filter list). The event count was down to 600:

The next step was to exclude events I knew weren’t related to the infection. Recognizing irrelevant events takes experience because it requires familiarity with typical Windows activity. For example, the first few hundred events of the remaining operations consisted of Explorer referencing values under the HKCU\Software\Microsoft\Windows\ShellNoRoam\BagsMRU registry key:

This key is where Explorer stores state for its windows, so I could exclude them. I did so by using Process Monitor’s “quick filters” feature: I right-clicked on one of the registry paths to bring up the quick filter context menu, and selected the Exclude filter:

Because I want to exclude any references to the key’s subkey’s or values, I opened the newly created filter, double-clicked on it to move it to the filter editor and changed “is” to “begins with”:

That reduced the event count to 450, which is a more reasonable number, but I saw still more events that I could exclude. The next set of events were the System process reading and writing registry hive files. Hive files store registry data, but it’s the registry operations themselves that are interesting, not the underlying reads and writes to the hive files. Excluding those reduced the event count to 350. I continued looking through the log, adding additional filters to exclude other extraneous events. After I was done filtering out all the background operations, the Filter dialog looked like this (some of the filters I added aren’t visible in the screenshot):

Now there were only 133 events and a quick glance through them confirmed that they were all probably related to Stuxnet. It was time to start deciphering them.

Stuxnet System Modifications

The first event in the remaining list shows Stuxnet, operating in the context of Explorer, apparently overwriting the first 4K of one of its two initial temporary files.

To verify that the write was indeed initiated by Stuxnet and not Explorer.exe, I double-clicked on the operation to open the Event Properties dialog and switched to the Stack page. The stack frame directly above the NtWriteFile API shows “<unknown>” as the Module name, which is Process Monitor’s indication that the stack address doesn’t lie in any of the DLLs loaded into the process:

If you are looking at stacks with third-party code you may also see <unknown> entries when the code doesn’t use standard calling conventions, because that interferes with the algorithm used by the stack tracing API on which Process Monitor relies. However, when I looked at Explorer’s address space with VMMap, I found a data region containing the unknown stack address 0x2FA24D5 that has both write and execution permissions, a telltale sign of virus-injected code:

The operations following those of Explorer.exe’s are those of an Lsass.exe process creating four files - ~Dfa.tmp, ~Dfb.tmp, ~Dfc.tmp and ~Dfd.tmp - in the account’s temporary directory. Many components in Windows create temporary files, so I had to verify that these were related to Stuxnet and not to standard Windows activity. A strong hint that Stuxnet was behind them is the fact that the process ID (PID) of the Lsass.exe process, 300, doesn’t match the PID of the system’s actual Lsass.exe process, which I identified in Part 1. In fact, the PID doesn’t match any of the three Lsass.exe processes that were running after the infection, confirming that it’s another rogue Lsass.exe process launched by Stuxnet.

To see how this Lsass.exe process relates to the others, I typed Ctrl+T to open the Process Monitor process treeview dialog (it can also be opened from the Tools menu). The process tree reveals that three additional Lsass.exe processes executed during the infection, including the one with a PID of 300. Their greyed icons in the treeview indicate that they exited before the Process Monitor capture stopped:

I now knew that this was a rogue Lsass.exe process, but I had to verify that these temporary files weren’t just created by routine Lsass.exe activity. Again, I looked at their stacks and saw the <unknown> module marker like I had seen in the Explorer.exe operation’s stack.

The next batch of entries in the trace are where things really get interesting, because we see Lsass.exe drop one of the two Stuxnet drivers, MRxCls.sys, in C:\Windows\System32\Drivers and create its corresponding registry keys:

I double-clicked on the WriteFile operation to see its stack and observed that the call to the CopyFileEx API meant that Stuxnet copied the driver’s contents from another file:

To see the file that served as the source of the copy, I temporarily disabled the write category exclusion filter by unchecking it in the filter dialog:

That revealed references to the ~DFD.tmp file that was created earlier, so I knew that file contained a copy of the driver:

A few operations later the System process loads Mrxcls.sys, activating the driver:

Next, Stuxnet prepares and loads its second driver, Mrxnet.sys. The trace shows Stuxnet writing the driver first to ~DFE.tmp, copying that file to the destination Mrxnet.sys file, and defining the Mrxnet.sys registry values:

A few operations later the System process loads the driver like it loaded Mrxcls.sys.

The final modifications made by the virus include the creation of four additional files in the C:\Windows\Inf directory: Oem7a.pnf, Mdmeric3.pnf, Mdmcpq3.pnf and Oem6c.pnf.  The file creations are visible together after I set a filter that includes only CreateFile operations:

PNF files are precompiled INF files and INF files are device driver installation information files. The C:\Windows\Inf directory stores a cache of these files and usually has a PNF file for each INF file. Unlike the other PNF files in the directory, there are no matching INF files matching the names of Stuxnet’s PNF files, but their names make them blend in with the other files in that directory. Like for the operations writing the driver files, the stacks of these operations also have references to CopyFileEx, and disabling the write-exclusion filter shows that their source files are also the temporary files Stuxnet initially created. Here you can see Stuxnet copying ~Dfa.dmp to Oem7a.pnf:

All of the writes to these files are performed by the Lsass.exe process with the exception of a few writes to Mdmcpq3.pnf by the infected Services.exe process:

When done with the copies, Stuxnet takes additional steps to make the files blend in by setting their timestamp to match those of other PNF files in the directory, which on the sample system is November 4, 2009. The SetBasicInformationFile operation here sets the create time on Oem7a.pnf:

Once Stuxnet has set the timestamps, it cleans up after itself by marking the temporary files it created for deletion when it closes them (the operations deleting the other temporary files are in other parts of the trace):

It’s odd that Stuxnet writes temporary files and then makes copies of them, but it doesn’t appear to be a significant aspect of its execution since no Stuxnet research summary even mentions the temporary files.

One operation in the trace that I can’t account for, and for which I’ve seen no explanation in any of the published Stuxnet analyses, is an attempt to delete a registry value named HKLM\System\CurrentControlSet\Services\Network\FailoverConfig:

That registry value and even the Network key referenced are not used by Windows or any component I could find. A search of the executables under the C:\Windows directory didn’t yield any hits. Perhaps Stuxnet creates the value under certain circumstances as a marker and this code automatically runs to delete it.

Next Steps

So far, our analysis of the Stuxnet infection with several Sysinternals tools has documented Stuxnet’s system impact at the time of infection, method of reactivation at subsequent boots, and provided a complete recipe for disabling and cleaning Stuxnet off a compromised system. In Part 3 I’ll wrap up my look at Stuxnet with the Sysinternals tools by examining how Stuxnet uses each of the four PNF files it created in order to gain some idea as to their purpose. I’ll also analyze a trace of a Windows 7 Stuxnet infection to show the method by which Stuxnet took advantage of a zero day vulnerability on Windows 7 (which has since been patched) to gain administrative rights when it was first activated with standard user rights. Continued with Part 3.

Mark Russinovich is a Technical Fellow on the Windows Azure team at Microsoft and is author of Windows Internals, The Windows Sysinternals Administrator’s Reference, and the cyberthriller Zero Day: A Novel. You can contact him at markruss@microsoft.com .

Comments

• Anonymous
  January 01, 2003
  @Michael Maybe, but I defintely don't see any sign of it being created in my examinations.

• Anonymous
  January 01, 2003
  @Fleet Command Thanks! A bunch of the cases and additional ones will be in the soon-to-be-published Sysinternals Administrator's Reference from Microsoft Press I'm coauthoring with Aaron Margosis.

• Anonymous
  January 01, 2003
  Well done, Mark. I just love reading these topics! (And I know how difficult writing them is!) Tell you what? Convert all these lovely blog posts into a hardcover; and you'll be surprised how fast it sells.

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  @CarlosG Bring the book to TechEd and I'll sign it there! @Guruprasad  Thanks for the feedback!

• Anonymous
  January 01, 2003
  @David Glad to hear that you loved the book! Please sumit a review to Amazon.com!

• Anonymous
  April 20, 2011
  Can't wait for Windows 7 part. Thanks again Mark!

• Anonymous
  April 20, 2011
  Can anyone tell me why Windows allows code segments that are both Executable and Writable? Seems to me this would probably eliminate most, if not all, viruses.

• Anonymous
  April 20, 2011
  The comment has been removed

• Anonymous
  April 20, 2011
  Can't wait to read the third part. As always, Mark delivers! Also, I just got your book !.. Hope to get it signed though :)

• Anonymous
  April 20, 2011
  Bravo! What a research man? Awesome! Good! I really Loved this Article! Would like to subscribe to some good RSS Feeder from you to stick on and digg a lot to read things! All the Best for your Future Contributions! :-) -- Regards Guruprasad balaji, http://about.me/GP_Me

• Anonymous
  April 20, 2011
  The comment has been removed

• Anonymous
  April 20, 2011
  Mark, this is both a fascinating read and a great tutorial on the use of your tools. Waiting for your book to arrive...

• Anonymous
  April 21, 2011
  @TanMan: See DEP and its associated compatibility problems.

• Anonymous
  April 21, 2011
  Way to go, Mark. I look forward to reading the Sysinternals Administrator's Reference.

• Anonymous
  April 21, 2011
  So would taking yourself out of the administrators group prevent this?

• Anonymous
  April 23, 2011
  The comment has been removed

• Anonymous
  April 24, 2011
  @AndrewRichards, @tam: prior to Microsoft patching the four 0day vulnerabilities that Stuxnet utilized, running as a "normal" user account would not have prevented the infection.

• Anonymous
  April 26, 2011
  The comment has been removed

• Anonymous
  April 27, 2011
  Mark, always informative! It's amazing how these little things can tell the story of a complex system. @Michael, read the Avira notes more carefully and you'll see that they say it's deleted too (They say it's a new value but with set "FailoverConfig"=-, which means deleted). @Kagamine You can always save the page.

• Anonymous
  April 27, 2011
  The comment has been removed

• Anonymous
  April 28, 2011
  Absolutely Amazing articles Mark!! This makes me feel that one doesnt need to be a developer to analyse malware behaviour. (I always thought otherwise... ) Simple existing tools like sysinternals could be used to analyse the behaviour in detail. But i still believe my efforts in learning and using IDA for disassembly would not go waste.. After all knowledge is always good. :). Looking forward to reading your novel ""Zero day"" Once its available in India. Once again. Great articles.  What i learnt from this article is not how to analyse the behaviour but how to actually think while analysing a malware and which tool to use fro getting the right information. Great thought process. Thanks Mark. You are a gem in the windows World. Looking forward to the next one in the series.. Cheers!!