The Case of the App Install Recorder
Adapted from the forthcoming book, Troubleshooting with the Windows® Sysinternals Tools, by Mark E. Russinovich and Aaron Margosis.
A customer had nearly a dozen software packages that wouldn’t install on Windows 7 x64. Every installation program failed immediately with an error message like the one shown in the screenshot below. However, they all installed successfully on 32-bit Windows 7. This error message usually indicates you’re trying to run a 16-bit program, which is not supported on 64-bit Windows versions:
You can use SigCheck to verify the image type. This screenshot shows a set of Seagate Crystal Reports 6.0 installers from 1997 that are 16-bit executables:
The packages all dated from the mid- to late-1990s. Although the packages installed 32-bit Windows components that could presumably run on 64-bit Windows, their installation programs were 16-bit as was often the case back then. 16-bit programs were the only kind that could run on all versions of Windows at the time, particularly on all the CPU architectures that Windows NT supported. Vendors of installer packages therefore used a 16-bit bootstrapper program to detect the operating system version and CPU architecture, and then install the correct binaries for that platform.
However, 64-bit Windows does not include the “NT Virtual DOS Machine” (NTVDM) emulator that enables 16-bit DOS and Windows programs to run on 32-bit versions of Windows NT and its successors, including Windows 7. Wow64 (Win32 emulation on 64-bit Windows) provides a limited ability to emulate some common 16-bit installers, but it didn’t help with these particular installers. Here’s a 16-bit installer running on 32-bit Windows 7. Note the old “Program Manager” icon in the taskbar, representing the Ntvdm.exe hosting process for 16-bit programs:
After installation, the 32-bit components worked fine on 32-bit Windows 7. It seemed likely that they would probably run fine on 64-bit Windows as well – if they could be installed. All that was needed was a way to replicate the installation steps that were performed on 32-bit Windows on 64-bit.
I came up with a way to record the installation on 32-bit with Procmon, save a filtered trace as XML, then process the XML with PowerShell scripts to capture the resulting file and registry modifications in a way that they could be copied to 64-bit. It identifies only the final names of objects that were moved or renamed, ignores temporary files and objects that were deleted before the installation completed, and excludes system changes made by processes not involved with the installation. The same idea can be used in any other scenario to capture any other types of file and registry key creations or modifications.
To begin, start Procmon, run the installation to completion, then stop the trace (Ctrl+E). Before beginning to apply filters, I recommend saving all events in the trace to a file using Procmon’s native file format as shown in the dialog box below so that you can come back to it later if you need additional data without having to run the installation again.
The next step is to apply filters so that the resulting trace shows only the file and registry operations of interest from the installation-related processes. Open the filter dialog box (Ctrl+L) and add an “Include” rule for each of these operations: CreateFile, WriteFile, SetRenameInformationFile, SetDispositionInformationFile, RegOpenKey, RegCreateKey, RegDeleteKey, RegRenameKey, RegSetValue and RegDeleteValue. Then add a criterion for “Result Is SUCCESS then Include”, as failed operations will not be of interest.
To filter on installation-related processes, open the Process Tree (Ctrl+T). Select the initial installation process as in the screenshot and click “Include Subtree” to set a filter for that Ntvdm.exe process and all its descendant processes.
You should also check to see whether the installation used any out-of-process DCOM components. Such components would run as child processes of the DcomLaunch service, which is hosted in the Svchost.exe instance started with the command line parameters “-k DcomLaunch”. You can inspect the command line of a process in the bottom of the Process Tree window by selecting the process in the tree (see the next screenshot). If any DCOM processes were started while the installation was running, select each and click “Include subtree”. Because it is also possible that an already-running DCOM process responded to a request from the installer, you could also select the DcomLaunch Svchost.exe and click “Include subtree” to include all DCOM processes, although doing so may pick up unrelated system changes.
Finally, if the installer created or modified any services or drivers via the Service Control Manager, the resulting registry changes will have been performed by Services.exe, so select it in the Process Tree and click “Include Process” (not “Include Subtree”). (System changes made by Services.exe may need to be inspected manually later to verify whether they should be captured and “played back”.)
Save the filtered trace to an XML file. Under “Events to save” in the Save To File dialog box, select “Events displayed using current filter” and deselect “Also include profiling events”. Under “Format”, select “Extensible Markup Language (XML)” and deselect “Include stack traces”. The next screenshot shows these options selected. As an option to reduce the size of the XML file, choose Options | Select Columns and show only the Operation, Path and Detail columns before saving the XML file. Save-as-XML saves only the column data selected for display, and those three are all that the script will need.
PowerShell is a particularly adept and flexible tool for manipulating XML, so I wrote a script to read the saved XML and build lists of the new and modified file system and registry objects resulting from the installation and then create a mirrored copy of the file system objects and a RegMods.reg file containing the registry changes that can be imported on another system. Portions of the script are described here; the full script is attached to this blog post and can be downloaded.
The script takes two parameters: the path to the Procmon XML trace, and the path to the target directory in which to build the mirror. For example:
PS C:\Installs> .\Capture-Recording.ps1 .\Crystal-Filtered.XML C:\Installs\Crystal
The script reads the input XML file, and inspects all the events in the trace in the order that they occurred:
# Convert input file into an XML document object
$inputFile = [xml](Get-Content $ProcmonXmlFile)
# Iterate through all the events in the trace
$inputFile.procmon.eventlist.event |
ForEach-Object {
As it processes each event element, the script saves the current element in the variable $ev:
# Save the current event as $ev
$ev = $_
It then looks at the event’s Operation and performs the appropriate action based on whether it is a CreateFile, WriteFile, etc.:
switch($ev.Operation) {
# File newly created (CreateFile may refer to "read" operations too)
"CreateFile" {
# perform actions
}
# Existing file modified
"WriteFile" {
# perform actions
}
# File rename - remove the old name, add the new name
"SetRenameInformationFile" {
# perform actions
Processing file system operations is fairly straightforward: for each creation or update event of a file or directory, it adds the event’s path, $ev.Path, to a sorted list of file system objects if the object’s path isn’t already in the list. Similarly, for each deletion event it removes the object path from the list if it’s in the list. Rename events are treated like a delete followed by a create: the old name is removed from the list and the new name added to the list. File system events are ignored if the path is in the user’s temporary directory or appears as a direct write to the user’s registry hive.
The one hitch is that we want to capture only file system modifications, and CreateFile events can be reads or writes. If the saved trace had filtered on “Category Is Write”, then read events would have been filtered out, but that isn’t possible because correct processing of registry operations needs read and write events, as I’ll explain shortly. It would be possible to look at $ev.Category had that column been added to the view prior to saving the XML. But the information we need is also in the Detail column:
# Verify whether this was a "write" operation
if ($ev.Detail.Contains("OpenResult: Created") -or
$ev.Detail.Contains("OpenResult: Overwritten")) {
The Detail column also provides the new object name on a rename operation (the old name is in $ev.Path):
$ix = $ev.Detail.IndexOf(" FileName: ")
$newName = $ev.Detail.Remove(0, $ix + 11)
And Detail also confirms whether a SetDispositionInformationFile operation is a file deletion:
if ($ev.Detail -eq "Delete: True") {
Processing registry events is a little more involved because registry value names can contain backslash characters, unlike the names of registry keys, files or directories which always treat backslashes as delimiters. Procmon captures registry paths as a single text value that can be just a key name or a key name plus a value name. In the latter case it’s hard to determine whether the last backslash is a delimiter between the key and value or part of the value.
To address this issue, the App Install Recorder script tracks all key “open” operations and not just “write” operations. A registry value cannot be accessed until its containing key has been opened, so the script maintains a list of all keys that have been opened, and then looks in that list for the open key whenever a value “write” operation (i.e., RegSetValue or RegDeleteValue) is processed. The script also keeps another sorted list of registry keys in which “write” operations were performed, and each item in that list has its own sorted list of the values that were created or modified within that key.
As with file system operations, “write” operations that create or update keys or values add to the corresponding lists, delete operations remove items from the lists, and renames combine deletes and additions. As with CreateFile, RegCreateKey can also be a read or write operation based on its Detail:
# A key was (potentially) created; add it to the list of known key names,
# and add it to the created-keys list if it was created.
"RegCreateKey" {
AddOpenKey($ev.Path)
if ($ev.Detail.Contains("Disposition: REG_CREATED_NEW_KEY")) {
AddCreatedKey($ev.Path)
}
}
After processing each of the events in the Procmon trace, the App Install Recorder script has built sorted lists of all resulting new and modified file system objects and registry data. It builds the mirrored copy of the file system results by iterating through the list of file system objects and copying them to the target location, retaining the directory hierarchy. Capturing the registry changes for playback is more involved. Here the script iterates through the sorted list of written registry keys and runs Reg.exe Export for each key, outputting to a temporary file. It then copies content in the file for the current key only to RegMods.reg, and only for those registry values that had been modified. (This probably isn’t the most efficient way to build a *.reg file, but it gets the results precisely the way Reg.exe Export produces them.) This screenshot shows the Capture-Recording.ps1 script running, with “The operation completed successfully” written every time Reg.exe was used.
When the script is done, it opens an Explorer folder window in the target directory. The next screenshot shows the mirrored file structure under the target directory and the RegMods.reg registration entries file. In this example, the script captured 1319 files in 76 directories and 1790 registry values in 1127 keys that were created or updated by the Seagate Crystal Reports 6.0 installation.
“Playing back” the App Install Recorder is as simple as copying the captured files onto the new system and running Reg.exe Import to import the captured registry data. When “playing back” a capture from a 32-bit system onto a 64-bit system, it’s important to ensure that 32-bit data ends up in the correct redirected locations. The AppInstallPlayback.cmd script shown in the next screenshot uses the 32-bit version of Reg.exe to import RegMods.reg so that keys are redirected to Wow6432Node as they would be for any 32-bit process. Similarly, it uses the 32-bit version of Xcopy.exe to copy the directory hierarchy, ensuring files are redirected to SysWOW64 when appropriate.
After creating an NTFS junction to “join” the installation directory to an equivalent name under “Program Files (x86)”, the Start Menu shortcuts shown here launched programs on 64-bit Windows that could not have been installed without Procmon.
Comments
Anonymous
September 07, 2014
Wow. sysWow, even. I'm awestruck by the detail I was unaware of, which explains why my earlier efforts sometimes came to nil. This is amazing. It is also sobering that a decade and a half into the 21st century, people like us still have to do things like this. It's like learning to oil a steam engine, but, hey, it still works!Anonymous
September 07, 2014
Agree very nice work ! But wouldn't it be better to move on to an alternative software solution ?Anonymous
September 09, 2014
When a company invests hundreds of thousands of hours into creating crystal reports with 16-bit software sometimes the level of effort to replace it is in the millions of dollars a solution like this is much more cost effective even if you spent a week doing it. [Aaron Margosis] True! To be clear, Crystal Reports in this case is 32-bit. The only 16-bit software here was the installation program.Anonymous
September 16, 2014
The scripts look awesome. I tend to use process explorer to make file and registry changes rather than making a whole new msi. But if the changes aren't simple it gets tedious.Anonymous
September 18, 2014
If those guys run software that old, they might also have an old version of SMS Installer 2.0. That one came with something called "Repackage Installation Wizard", which would automatically pick up all the changes to the system during installation and then even create an executable. But then again, I guess the point here is that one can use the Sysinternal tools for all kinds of things one usually would need to use 3rd party software for. On the other hand, Procmon alone doesn't really help you here; unless you have your own Powershell scripting skills or one of the nice Sysinternals people just so happens to have the same issue and hands out the source.Anonymous
November 03, 2014
Maybe I can use this to package all the changes from the 46 active setup processes that run for each new user!Anonymous
November 04, 2014
Ack. I don't think it supports HKCU? Warning: these items not found -- HKCUSoftwareMicrosoftMediaPlayerPreferencesAppColorLight HKCUSoftwareMicrosoftMediaPlayerPreferencesAppColorMedium HKCUSoftwareMicrosoftMediaPlayerPreferencesAppColorDark [Aaron Margosis] Make sure to run the script in the same security context in which you ran Procmon. The definition of "HKCU" depends on which user is logged on when it's evaluated.Anonymous
November 05, 2014
My bad. I thought I could just filter "Category is Write", but your script seems to require "RegOpenKey". Works great! [Aaron Margosis] Yes, and when I demoed the predecessor of this script at TechEd I was using "Category Is Write". That turned out to be insufficient to get the necessary context to reconstruct the registry operations, as the blog post explains.Anonymous
November 05, 2014
It might be a good update for process monitor to treat registry keys and values as separate things, at least in the xml logs.Anonymous
February 21, 2015
Brilliant and well detailed article Aaron. Nice work!Anonymous
June 30, 2015
Hmm, I was seeing if "category is write" would filter the same thing, except regopenkey. I couldn't create a "regopenkey" operation using regedit interactively, but I could with a .reg file. I'm not sure how to invoke "regrenamekey", even in powershell.Anonymous
August 26, 2015
I stumbled across this article when searching a way to record an entire software installation so that I could then script that process. Luckily for me this article, the technical error of a 16-Bit application on x64, was another technical issue I was working on too. I followed the article and for the most part did not have trouble with obtaining the ProcMon trace and running the Powershell script. My issue(s) however came in at the running of the batch file. The article does not go into detail regarding the running of it - I was consistently getting "file not found". I modified the script to directly define the folder structure to copy. The regMod import was also 99% successful but at the end did mention something about an error - not found or something related. Another problem I encountered was creating the junction point - I ran the following command: mklink /J "C:APPName" "C:Program Files (x86)APPNameDifferent" - it did create a junction. At this point the registry data and the files have copied successfully. Next I was unaware of how the all programs shortcuts to the applications plus icons got there but ... looking at my executable file that runs on 32-bit (just a simple EXE) I went to my 64-bit computer and ran the same EXE but it did not open. It prompted the same error - version is not compatible. Any ideas what I could have done wrong? The set up installation was successfully performed including the capturing of it - to the best of my abilities. (And the application does work on 32-bit).Anonymous
November 16, 2015
Note that procmon logs currently truncate reg_binary values to 16 bytes. [Aaron Margosis] I don't think that's an issue because the script isn't trying to get the registry data from the logs, just which values need to be retrieved.Anonymous
March 16, 2016
Wow Aaron, great job!Anonymous
November 16, 2018
The comment has been removed