• Logonui.exe (logon dialog is presented to use)
• Userinit.exe (started after a user has entered credentials, is successfully authenticated, and subsequent processes are started for that user to set up their desktop)
• Explorer.exe (start shell)
• Process Monitor (the user runs Process Monitor to stop and save the trace, which also indicates the desktop is at or near a "ready to use" state)

Windows Reboot and Logon Analysis Tool

Sysinternals’ Process Monitor

Recording a “Reboot Cycle” trace, using the Sysinternals’ Process Monitor

1. Start Process Monitor (this may require elevation)
2. The ‘Process Monitor Filter’ interface will be displayed. Click the ‘Reset’ button to reset filters to default values, and then click the ‘Ok’ button.
3. Click the ‘Capture’ button to stop the current real-time trace, as depicted below:

1. Before the boot logging is started, some filtering can be enabled that will reduce the captured trace to a very small size, and filter out events that are not needed to analyze reboot and logon.
   1. On the ProcMon icon bar, click to de-select the following categories of events:
      1. Network (name is “Show Network Events”)
      2. Disk and File (name is “Show File System Activity”)
      3. Registry (Show Registry Activity)
   2. Now, click ‘Filter’ from the ProcMon menu, and then click the menu item named ‘Drop Filtered Events’.
2. In the Process Monitor menu, click ‘Options’, and then click ‘Enable Boot Logging’, which enables the Process Monitor boot logging, until the tracing is stopped.
3. An option dialog will appear titled ‘Enable Boot Logging’, which offers an option to “generate thread profiling events”. Don’t click any option, just click the ‘Ok’ button to enable boot logging on the subsequent reboot.
4. After clicking the "Ok" button, close Process Monitor and restart the computer.

Analyzing a “Reboot Cycle” trace, using the Sysinternals’ Process Monitor

1. After rebooting the device, logon to the device, and then start Sysinternals’ Process Monitor.
2. You may be prompted to reset the filter, which you can do, and then click OK.
3. You will then be prompted to save the current trace. You can click ‘Ok’ and then choose the folder to save the trace to.
4. Once the trace is saved, the boot trace will be available and displayed in Process Monitor.
5. Click ‘Tools’ and then click ‘Process Tree’.
6. Click the very first item in the far-left column, which should be the pseudo-process ‘Idle’, and note the clock time.
7. Next, in the ‘Process Tree’ display, scroll down while looking in the far-left column for the item ‘Logonui.exe’. When found, click ‘Logonui.exe’, and again note the clock time.

• At this point in computer startup, the user entered credentials and pressed ENTER. The next marker in the trace to look for is the process ‘Userinit.exe’. ‘Userinit.exe is the process that is launched if the user’s credentials are verified, and initiates the subsequent chain of events leading to the user’s shell starting, desktop starting, and the important marker “desktop ready to use”.

• The ‘Userinit.exe’ process should be relative close but under’ the previously noted process ‘Logonui.exe. Note the clock time for starting of the ‘Userinit.exe’ process.
• A good indicator of “desktop ready to use”, in this case, is the starting of the process ‘Procmon.exe’. Recall that after logging on, we started ‘Procmon.exe’ in order to stop and save the boot trace. Note the clock time of starting ‘Procmon.exe’.
• The difference in clock time between starting of ‘Userinit.exe’ and ‘Procmon.exe’ is roughly that particular user's overall logon time.

Getting Ready for Windows 10: Servicing Windows Client and Server in a Managed Environment Today

If you care about the topic of Windows updating/servicing, this is the presentation for you.  This presentation is all about current Windows (client & server), but NOT Windows 10.  There are some topics discussed for Windows 7 and Windows Server 2008 R2.  Windows 10 strategies were covered at Ignite in other presentations.

NOTE: I want to point out that I am NOT a member of the Windows PG, I am a Platforms PFE in Microsoft Premier Services.  All of the information in this blog post, is what was discussed in the Ignite presentation.  The OFFICIAL bottom line on all these topics will come from the authoritative blog: Windows for IT Pros.

IMPORTANT:  None of this information changes any guidance on security updates.  Security updates should be applied as soon as possible, within existing change management processes for the enterprise. Microsoft guidance on security updates and other related topics is available at this location:

Microsoft Safety and Security Center

Windows Updates: Categorization Processes

• Since Windows 8, Microsoft has released updates that are NOT categorized as "Critical" or "Important", those updates are released with status "Optional".  This included most, if not all of the previous monthly Windows 8x/Server 2012x rollups.
• Optional updates are discovered and installed by a relatively small percentage of users around the world that are Windows Update clients.
• Keeping the updates as Optional (at first) provides time to discover and correct problems before those updates are promoted to status Recommended or Important.
• Microsoft would like to have more customers not only try Optional updates as soon as possible, they would like customers to opt-in to the Customer Experience Improvement Program (CEIP) so their computing devices can send telemetry back to Microsoft, including how those update installations went.
• Having IT Pros and enterprise customers sending telemetry on Windows updates, further aids in evaluating overall product quality with patches.  The enterprise segments are very important with Windows updates because the enterprise environments have different environments than consumer devices, and thus can help uncover issues that may not be discovered otherwise until deployment.

Hotfix Deployment Guidance: Then and Now

• The text in today’s hotfix KB articles says don't apply this hotfix unless you experience this particular problem. Microsoft is changing the guidance on hotfixes and Optional updates to say “don’t wait to experience blue-screens, hangs, or data corruptions issues if there is a fix available that could correct these today”.  Proactively evaluate available fixes, whether Optional or hotfixes, as those fixes are in fact tested more stringently than in the past. Problems such as bugchecks, hangs, or data corruption, are not problems you should wait to experience.
• Problems such as bugchecks, hangs, or data corruption, are not problems you should wait to experience.
• Microsoft is changing guidance to say, "Deploy Hotfixes Proactively", especially bugchecks, hangs, and data corruption, maintaining strict testing standards, etc.
• Microsoft will be updating hotfix KB articles to reflect Microsoft's updated Windows updating/servicing guidance.

Optional Update Guidance & Other

• Microsoft would like to get telemetry from Optional updates from IT Pros which will help update quality and be a determinate for promotion to Recommended or Important.
• Once an update appears in WU as "Recommended", that particular fix has been installed/deployed to millions of Windows devices already, so has been vetted to some degree.
• Going forward, Windows updates listed as Recommended, Optional, or Important will be published as "one fix, one package".
• Microsoft wants all customers to proactively install updates to help overall product quality.
• The bottom line: Microsoft would like customers to proactively install available updates, not just security updates.  For the enterprise, this would mean introducing Optional updates into the change control process as soon as they are released, for eventual rollout to the production computer systems.

What about those Rollups?

• Rollups are a single package with multiple fixes
• Up until December 2014, Windows 8x/2012x had "monthly rollups".  At least for now, no more monthly rollups.
• There may be from time-to-time., cumulative "convenience rollups" (not a technical term).  These provide a way to get current with all fixes by applying one package.
• Moving forward, HOTFIXES will be published to the Windows Update Catalog (no timeline specified).  Therefore these can be relatively easily ingested into Windows Server Update Services, for enterprise deployment.
• There WILL BE A ROLLUP FOR WINDOWS 7 & WINDOWS SERVER 2008 R2 (no timeline specified).

That last part if fantastic for those still running Windows 7 and/or Windows Server 2008 R2.

Thanks for reading!

Robert M. Smith, Senior PFE
Microsoft Premier Services

Part 1: Getting Started with Debugging Symbols

Hey Folks.  Platform PFEs Robert Smith and Dan Cuomo here to give you an overview on debugging with “symbols.”  Now if you’re thinking to yourself, “I have no use for an ancient Egyptian writing form” or even “I’m pretty sure that cornfield thing was a hoax” then this article is for you.

In this series of posts, we’ll give you an overview of what debugging symbols are, why you need them, and how to get them.  We’ll cover how to use symbols with the Debugging Tools for Windows, as well as other tools such as Windows Performance Analyzer (WPA) and the SysInternals suite.  If you operate within the challenging confines of an isolated/disconnected (AKA air-gap) network, we’ll show you how to setup your own symbol resources in that environment.  In addition, we’ll give you some tips to managing your symbols and also provide some troubleshooting advice for common issues you might run into.

In this first post, we’ll cover what symbols are and why they’re pertinent to you and provide an example to get you started.  We would like to mention first that there is an existing article on this topic that is also a great reference:

Dump Analysis in a Disconnected Environment

We will add some information and a few tips and tricks along the way.

What are symbols and why do I need them?

The typical field scenario for symbols occurs during the analysis of crash-dumps copied from other workstations.  While loading crash-dumps, the debugger will parse the “loaded module list” of the process or dump file.  Without symbols, the debugger might have incomplete or inaccurate information.  Here is an example from the WinDbg debugging tool showing a call stack before symbols are loaded.

Before symbols have loaded:

In the absence of a correct symbol for a given module, the debugger will still try to locate and display the parent function for a given call-stack, such as the one above.  If you are debugging on the same machine the dump originated from, the debugger will attempt to utilize the same named module (file) from your local file system to assist in call-stack display.  The problem comes when you debug from a different platform, as is the case here.  The dump above was taken from a Windows 7 system, and is being analyzed on a Windows 8 system.  There are modules with the same name, but different versions.  Therefore, you cannot trust the output you see, even if it looks valid.

We can fix our symbols path(s) by typing “.sympath <path>” in the debugger.  You can follow that by .reload and the debugger will try again to retrieve and load the correct debugging symbols:

Notice the function calls highlighted.  The top-most in particular, is now the correct function.

Debugging symbols provide “symbolic” information about software, such as:

• Function names and the addresses of their entry points
• Variable definitions (global and local)
• Frame pointer omission (FPO) records
• Source code filename and line-number

Though there are different types of symbols, .PDB is a very common type and the type currently available for debugging Windows.

During compilation of “retail” code, the symbolic information about code can be separated from the modules, and a “program database” (PDB) is created as a result.  The PDB files are matched to the original binary via a GUID, placed in .DLLs, .EXEs, and the .PDB files.  Matching the proper symbol to the associated binary is vital for accurate output from the various tools that utilize debugging symbols for analysis.  Here is an example from the WinDbg debugging tool showing a call stack after symbols have loaded.

General Symbol Location

This is a high level overview of how the debugging engine (Dbghelp.dll), locates and make symbols available to the debugging tools:

1. Set up “symbol path” based on whether or not a path is currently defined.  If no path is defined, symbols are searched for in a pre-defined list of well-known symbol locations.
2. Determine GUID of given module
3. As a given call-stack is parsed, search for symbols matching modules referenced in stack.
4. Search local paths first.  If no local matching symbol located:
   1. Connect to symbol server, if available, send request to server
   2. If a match is found, download and load the symbol
   3. If a local cache is defined, add that symbol to the local cache, so subsequent symbols requests can be satisfied locally instead of over a network
   4. If no matching symbol can be found, display an error and help text

Using the Microsoft Public Symbol Server

Microsoft provides public access to a symbol server that contains symbols for Windows operating systems and some other components such as the .Net Framework. This catalog of symbols is not guaranteed to be complete, but it is extensive and is almost always up to date.

If you have access to the Internet during debugging, you can configure the debugger (as you’ll see in the next section) to download symbols as needed during a debugging session, rather than downloading symbol files separately before a debugging session. The symbols are downloaded to a directory location that you can specify and loaded by the debugger locally.

The URL for the Microsoft symbol store is https://msdl.microsoft.com/download/symbols.

Getting Started: Debugging a Local Process

WinDbg.exe is the name of the executable that starts the GUI based debugger from the Debugging Tools for Windows package.

After you’ve installed the Debugging Tools for Windows, browse to the start location of your Debugging Tools, which you can find in the Start Menu.

To get started, let’s debug a process on the local computer and add symbols to enhance analysis.

Open WinDbg.exe -> on the menu bar click "File" -> then click "Open Executable" > browse to:

%windir%\System32\net.exe

At the prompt, type .sympath srv* to tell the debugger to use the default symbol server path and retrieve symbols from the default symbol store; a structured folder path of symbols.

As you can see in the highlighted section above, the default path includes the Microsoft Public Symbol Server covered in the previous section

Now type .reload

Lastly type x net!

Symbols for the critical modules will be download to the local symbol cache as you’ll see in the next section.  You’ll also find a pingme.txt file in that same location.  This file is auto-generated and indicates that the location is a symbol store, even if the SRV* prefix is not provided.  On our test systems, the default cache locations were:

• Windows 8 - \ProgramData\dbg\sym
• Server 2012 - \Program Files (x86)\Windows Kits\8.1\Debuggers\x64\sym

Caching symbols

As symbols are resolved by the debugging tools engine, they are cached either temporarily, or long-term.  The behavior is determined by the syntax of the symbol path.

The following example shows how to set a symbol path, using what is referred to as a “downstream store.”  The debugging engine copies the symbol into a subfolder in the “downstream store” after retrieving that file from the symbol server.

In subsequent analyses, the symbol in the “downstream store” will be utilized first removing the need to browse the remote store.  In the example below, the Microsoft Public Symbol server will be used to find symbols if an appropriate symbol cannot be found in C:\DownStreamStore.  If found in the Microsoft Public Symbol Server, the symbol will be cached in the C:\DownstreamStore for future use.

NOTE: Substitute your downstream store path for c:\DownstreamStore:

srv*c:\DownstreamStore*https://msdl.microsoft.com/download/symbols

In field, and especially when debugging disconnected networks, you may not have the luxury of connecting to the public symbol server to retrieve symbols on demand.  Instead you can take a copy of your “downstream store” folder and subfolders to your isolated network, and use that as the basis for your own symbol server. 

In this post we covered what symbols are, why they’re pertinent to you, and provided an example to get you started in the debugger.  In part two we’ll cover how to setup your own symbol server including how to retrieve symbols for an offline environment and setup your symbol path.

Thanks for reading,

Robert Smith and Dan Cuomo

If you’re interested in reading the other parts of this series, please see one of the links below.

• Part 1: Getting Started with Symbols

• Part 2: Setting up your Symbol Server and Setting the Symbol path

• Part 3: Managing your Symbols

• Part 4: Using Symbols with Tools

• Part 5: Troubleshooting Symbols

• Complementary: Step-by-Step - Debugging Tools for Windows Installation

• Monthly rollup
• Globally released update
• Limited released update

Monthly Rollup

Windows 8x and Windows Server 2012x were being serviced mainly through monthly rollups, which usually correspond to the time when security updates are released.  For the most part, these rollups were NOT cumulative since the last milestone.  They were essentially cumulative since the previous rollup, with an occasional exception (April 2014 & November 2014).

NOTE:  The monthly rollups are sometimes listed as "Optional" or "Recommended", but should be evaluated and applied as soon as practical.  These rollups often contain fixes to address performance, reliability, feature, or role updates (ex. Hyper-V, Failover Clustering, etc.).

Our "ASKPFEPLAT" group has a great blog post on this topic:

Update Rollups For Windows Server 2012 and Windows 8 Explained

NOTE: Per the Ignite 2015 conference, and this presentation specifically, Microsoft has announced that there will be no more monthly rollups.  There may be an occasional "convenience" rollup.

Global Distribution Release (GDR) Update

GDR updates are available through Windows Update, Microsoft Update Catalog, or Windows Server Update Services (WSUS).  GDR updates are usually localized in various languages, and are tested as widely as possible before public release.  In the past we might have seen these show up in a list of fixes available on "Patch Tuesday".  With the rollup mechanism being used in Window 8x/2012x, there are likely to be fewer standalone updates than previous operating systems.

Limited Distribution Released (LDR) Update

Limited released patches, also known as "hotfixes", "QFEs", or "LDR", are fixes that are generated most often as the result of a critical customer support incident.  These fixes generally need to be created, tested, and delivered to customers in a condensed timeline.  Limited release fixes that can be discovered through a published KB article, and obtained by the public, are fully supported by Microsoft. These types of fixes are available for public download, but usually only available through direct download, not through Windows Update or the Windows Update Catalog.

LDR fixes were historically intended to be included in the next milestone, whether that is service pack or new OS version.  There is a great blog post on the "NTDebugging" blog that goes into this topic in greater detail:

Windows Hotfixes and Updates - How do they work?

"Unofficial" Operating System Patching Guidance (Windows 8x/Server 2012 Rx)

PLEASE NOTE: 

I am offering this guidance, to aid in keeping operating systems as up to date as possible, with all fixes, not just security, and with as little effort as possible.  I provide these recommendations in the context of a Microsoft PFE that works on site with customers, and experiences the various challenges of Windows servicing in general.  I am making no official guidance statements with regard to Windows servicing, only my general guidance for enterprise customers.  Any official guidance on Windows servicing will be posted on a top level Microsoft blog site, such as the Windows Blog

• Install all rollups available since the previous "milestone".  A milestone would be from Windows 8 to Windows 8.1, or from Windows Server 2012 to Windows Server 2012 R2.
  
  NOTE: One rollup in particular, KB2919355, is cumulative since the Windows 8.1 and Windows Server 2012 R2 respective milestones.  This rollup IS cumulative since those milestones, and should be applied first after a new OS installation.
  
  
  • KB2919355, "Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 update: April 2014", has a prerequisite as well; 2919442, "A servicing stack update is available for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2: March 2014".  If you have installed your operating system some number of months after 2919442 was released, and have installed all offered updates, then you may receive a message that 2919442 is not applicable.  This may happen if you have a later version of KB2919442 installed, such as KB2989647.  Related error codes are 2149842967, or 0x80240017h, and translate as WU_E_NOT_APPLICABLE (wuerror.h).
    
    
  • If KB2919355 is installed manually, there are 7 files in all to download, and must be installed in a specific order.  From the installation notes of KB2919355 download page: "These KB's must be installed in the following order: clearcompressionflag.exe, KB2919355, KB2932046, KB2959977, KB2937592, KB2938439, and KB2934018.
    
    
  • KB2919355 has a "Known Issues" section for both pre and post installation issues.  It is important to review the Known Issues section of this, and other KBs, before proceeding.
    
    
    
• If available, use Windows Update, or Windows Server Update Services (WSUS).  Windows Update, and Automatic Updates in Windows, will evaluate what updates are currently installed on the operating system, what updates are available, comb the list, evaluate superseded patches, and offer the most condensed list of currently available updates.
  
  
• Critical Updates should be tested and installed as soon as possible with high priority.
  NOTE: This blog post does NOT change any existing security guidance.
  
  
• Important updates should be tested as soon as practical, and installed as soon as practical.  The "Important" class of updates can address security, performance, or reliability issues.
  
  
• Recommended updates should be reviewed, tested, and installed as applicable.  The monthly rollups are sometimes categorized as "Optional", such as the October 2014 Rollup.  These rollups should be evaluated as soon as possible, as they often contain fixes for components in Windows Server Features and Roles.  The default setting for Windows 8x/2012x is to install Recommended updates the same as Important updates.  For systems that are serviced through Windows Update, this setting will ensure that nearly all available updates are installed and those systems are running up-to-date code.
  
  
• Optional updates should be reviewed, and installed possibly in a pilot group.  The Optional updates are either just published new updates, or updates that don't fall into one of the other categories.  Per the Ignite 2015 conference, Optional updates are evaluated periodically and their category ranking could be promoted to Recommended.  There is no setting available in the Windows Update client that will automatically install Optional updates.  Optional updates have to be installed manually, or through some other means such as SCCM.

The following page has good general patching guidance with respect to security updates:

Stay up-to-date for more secure web browsing

There is also a security guide available that goes into some of the topics discussed in this blog post, in great detail:

Microsoft Security Update Guide, Second Edition

The Windows Update component makes it simple to install all available updates automatically.  If all settings are set to "automatic", the only user intervention would be to reboot when notified, which is very important.  Windows will prompt when a reboot is needed, and will automatically reboot after a period of several days.  The reason is that patches may contain updated components that are "in-use", and cannot be replaced while the operating system is running.  These components are "staged" to be replaced during the next full computer restart.  During the time that there are pending reboots necessary, some functions in Windows are not available, such as adding or removing Roles or Features, and/or installing further operating system patches.  If you attempt to perform one of these operations, you will likely receive a prompt to restart your computer first, to handle to the pending changes.

This guidance is subject to change in the current releases, or in future releases.  This blog post makes no recommendation to change any setting or practice with respect to operating system security or hardening.

Thank you and safe, reliable computing.

Robert M. Smith, Sr. PFE, Platforms

Windows 2008 R2 Performance and Hotfix Recommendations 

Hello from the field.  It's been a little since my last post.  I spend quite a bit of time analyzing performance on Windows Server.  I have recently put together a list of fixes and settings that I recommend to any person running Windows Server 2008 R2.  These fixes are to maximize performance and stability.  Please note that the recommendations in this blog post are not necessarily required.  My intent is not to change any current guidance by Microsoft Corporation with respect to security or privacy.  I am only seeking to increase awareness of operating system patches that are available, but mostly not offered proactively in any form.

First and foremost is a rollup for Windows 7 and Windows Server 2008 R2: 2775511.  This rollup is recommended, per the verbiage in the article.  Please note that there are several patches that should be applied along with the rollup, and are all documented in the KB article.

This post covers mainly a category of operating system fixes known as "hotfixes", or "LDR" fixes.  Microsoft creates hotfixes as a result of either a critical issue reported, issue identified, and a code fix created, or as a result of "Watson" telemetry of Windows computers around the world reporting errors in Windows until some threshold is crossed.  Here is a recent TechNet article on Windows Telemetry:

What's New in Telemetry
https://technet.microsoft.com/en-us/library/jj651022.aspx

I do not advocate installing hotfixes just because they are there and the latest code.  I do advocate occasionally evaluating what hotfixes are available that address significant performance or reliability issues.  Which brings me to performance.

Power Policy Performance Considerations

Windows has long provided means to help conserve power.  This makes sense for many situations.  In the case of servers, and especially always busy servers like Exchange or SQL, there are documented issues with the default Windows power policy which is balanced.  Changing the Windows power policy to High Performance often results in better overall server performance.  The increased performance may come at the expense of increased power usage and heat generation, so that is a factor that has to be considered, especially in a dense datacenter environment, or an environment where power and cooling are already an issue.

Note that you can utilize the built-in Powercfg.exe to change power policy at command line, which means it can be scripted.  A scheduled task can be created to set the power policy to High Performance just before peak hours, and set it back to Balanced just after peak hours.  That might be an acceptable compromise of power savings and performance gains.

The power policy topic is well documented and you can find more about that here:

Degraded overall performance on Windows Server 2008 R2
https://support.microsoft.com/kb/2207548

Windows Server 2008 R2 Hotfixes Available

I would like to reiterate, this is only a list of available fixes for Windows Server 2008 R2, and all are released since rollup 2775511.  Any Windows fix should be first carefully evaluated and tested prior to deployment in production.  Microsoft does not test "Limited Distribution Release" (LDR) Windows fixes as thoroughly as "General Distribution Release" (GDR) fixes.  LDR and GDR are Microsoft terms to categorize Windows fixes.  GDR fixes are international release, widely-tested fixes, whereas LDR fixes are not widely tested.  LDR fixes are not unsupported however.  You won't put your server in an unsupported state by applying LDR fixes.  And often times if you have to engage Microsoft CSS for troubleshooting you will be given advice to install a recent LDR fix for a given component to ensure the issue being addressed is not the result of a fix that already exists.  You can find more about Windows servicing technology with these searches:

Bing "LDR" "GDR" site:microsoft.com
https://www.bing.com/search?q=ldr+gdr+site%3Amicrosoft.com&qs=n&form=QBRE&pq=ldr+gdr+site%3Amicrosoft.com&sc=0-8&sp=-1&sk=

The Windows Servicing Guy Blog:
https://www.bing.com/search?q=ldr+gdr+site%3Amicrosoft.com&qs=n&form=QBRE&pq=ldr+gdr+site%3Amicrosoft.com&sc=0-8&sp=-1&sk=

Also, please note that this list does not change any existing security guidance from Microsoft.  Any security related fixes available should immediately be evaluated, tested, and deployed per existing Microsoft guidance from our Security teams.  The settings and fixes in this blog post are intended to address performance and reliability issues.

Hotfixes available for Windows Server 2008 R2, post SP-1:

By the way, fantastic work for all those involved, and you know who you are, that pushed to get that Post-SP1 rollup out.  That rollup can be applied to Windows 7 as well as Windows Server 2008 R2.  If nothing else, I highly recommend 2775511 be evaluated as an across-the-board standard for Windows 7 or Windows Server 2008 R2.  There will continue to be fixes for Windows 7 and Windows Server 2008 R2 until they reach end of support, so I may append the list above as I come across new fixes that are very important.

Thank you for reading.

Sincerely,

Robert M. Smith
Senior Premier Field Engineer, Microsoft

Performance problems, especially as they may relate to the storage subsystem, can be quite difficult to troubleshoot.  Enterprise storage technology has come a long way since the SCSI controller with an array of disks.  Fortunately, there are some great tools available to help narrow down where to look more closely for storage performance problems.  This blog post covers the Windows Performance Analysis Toolkit (WPT), as used for analyzing performance in the storage subsystem.

The facility that enables the analysis I am about to cover is called "Event Tracing for Windows" (ETW).  The Performance Analyzer is built on top of the ETW infrastructure. ETW enables Windows and applications to efficiently generate events, which can be enabled and disabled at any time without requiring system or process restarts. ETW collects requested kernel events and saves them to one or more files referred to as "trace files" or "traces." These kernel events provide extensive details about the operation of the system. Some of the most important and useful kernel events available for capture and analysis are context switches, interrupts, deferred procedure calls, process and thread creation and destruction, disk I/Os, hard faults, processor P-State transitions, and registry operations, though there are many others.

One of the great features of ETW, supported in WPT, is the support of symbol decoding, sample profiling, and capture of call stacks on kernel events. These features provide very rich and detailed views into the system operation. WPT also supports automated performance analysis. Specifically, xperf is designed for scripting from the command line and can be employed in automated performance gating infrastructures (it is the core of Windows PerfGates). xperf can also dump the trace data to an ANSI text file, which allows you to write your own trace processing tools that can look for performance problems and regressions from previous tests.

The following information will be mostly about the WPT tool called "Xperf.exe".  Xperf.exe is the command line tool used to start, stop, and manage traces.  The usage of Xperf.exe is documented thoroughly in the help file included with the WPT titled "WindowsPerformanceToolkit.chm".

Obtaining the WPT Tools

The WPT comes with the Windows Software Development Kit (SDK), which is a free download

Microsoft Windows SDK for Windows 7 and .NET Framework 4
https://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8279

You can install just the WPT from the SDK using the custom installation option.  You can also choose to just download the WPT as a Windows Installer file (.MSI) in case you need to install that later on other machines.  The .MSI packages are delivered under the section called "Redistributable" components.  There will be 3 .MSI files, one per machine type architecture (x64, IA64, and x86).

One important thing to note about the WPT.  You can capture a trace from a running production computer without making any changes to that computer.  That is, you do not need to install the SDK, nor the .MSI package.  You can install the WPT on a different machine, even a virtual machine.  Then you can just copy the files, folder, and sub-folders to the production machine.  In fact, you really don't need to copy all the files to the production machine, just the analysis machine.  The following blog has documented that you need just two application files; Xperf.exe and Perfctrl.dll:

Xperf Articles:
https://blogs.msdn.com/b/pigscanfly/archive/2008/03/15/xperf-articles.aspx

More About the WPT

Windows Performance Analysis Tools
https://msdn.microsoft.com/en-us/performance/cc825801.aspx

WPT and XPerf utilize Event Tracing for Windows to record system activity that can later be used for performance analysis.  Here is more about Event Tracing for Windows:

About Event Tracing
https://msdn.microsoft.com/en-us/windows/hardware/gg487334

CL16: Performance Analysis Using the Windows Performance Toolkit
https://ecn.channel9.msdn.com/o9/pdc09/ppt/CL16.pptx

In fact, the blog above has great information about getting started.  The only other real "best practice" would be to use the same version of Xperf for the respective architecture (x64 for x64, x86 for 32-bit, IA-64 for Itanium).

Getting Started: Capturing Storage Performance Data

Scenarios

Here are scenarios that you would use the WPT:

• Start the trace from a batch file, based on anticipated load.  For example, the storage seems sluggish at a particular time, so a trace is started at that time, run for some period of time, and then stopped.
• Run a trace in "circular mode", which means the trace runs all the time, but is limited to a pre-determined size.  The pre-determined size might be 1 GB, which on most system allows a good time "window", though the size is impossible to predict.  1 GB might be 30 minutes on 1 system or 10 on another, depending on number of Logical Units (LUs), workload, etc.  And, you can certainly make the trace larger, just keep in mind the analysis machine and the time it takes to load and process the resulting trace file.

Considerations for Starting a Trace

Once you get the files copied to the target server, or the WPT installed, you can start a trace easily by going to an "administrator" .CMD prompt and running:

If you run that command and get no feedback, you are capturing a trace.  You will only get feedback if there is an error or if a "kernel" trace is already running, as is the case with the following example:

From this point, the trace captures to buffers in memory and periodically flushes the trace information to a disk file.  One thing to keep in mind is that the Xperf process does have a small amount of overhead of RAM for buffers, CPU for processing the trace events, and storage performance and space for the target disk device.  The output from Xperf can be directed to a particular location.  Generally you are safe to start  your trace in the same volume as your operating system is installed, or some other volume other than the one being analyzed.  You can start your trace on a different drive, or use the "-f <location>" parameter to redirect the WPT output.

NOTE:  There are many different components that you can trace using Xperf.  For storage performance analysis, here are some suggested ways to start Xperf, along with the charts available in the resulting traces:

xperf -on FileIO+Latency+DISK_IO+DISK_IO_INIT+SPLIT_IO

Xperf -on FileIO+DISK_IO+DISK_IO_INIT+SPLIT_IO

The differences in the above commands are the result of parameters passed in to Xperf when starting a trace.  There are many different parameters that can be used to start a trace.  The parameter above, “diag”, is a group (kernel group) of commonly used parameters.  Here is the list of kernel groups available with the 7.1 WPT version of Xperf:

Kernel Groups:

       Base           : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+PROFILE+MEMINFO
       Diag           : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER+COMPACT_CSWITCH
       DiagEasy       : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER
       Latency        : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PROFILE
       FileIO         : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+FILE_IO+FILE_IO_INIT
       IOTrace        : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+CSWITCH
       ResumeTrace    : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+PROFILE+POWER
       SysProf        : PROC_THREAD+LOADER+PROFILE
       Network        : PROC_THREAD+LOADER+NETWORKTRACE

The main advantage of gathering one set of parameters versus another is that you have more information to drill down into by adding the “latency” kernel group flag.  The down-side will be larger overall trace size, and a bit more overhead on the system being measured as the traces are running.

Stopping a Trace

When you are ready to stop a trace, you can easily stop with the "-stop" parameter of Xperf.exe.  In nearly all cases however, I would recommend using the "-d" parameter to stop (and merge) a trace.  The only time you wouldn't be concerned would be if the trace and analysis computer were the same computer.

The difference between the two parameters is that "-stop" simply stops the capture and flushes everything to disk, while "-d" does the same but merges additional information about the current system into that trace.  The benefit of using "-d" is that the trace is now portable, meaning you can copy that to a different machine and perform analysis there.  This is what happens most of the time with production systems, because we usually don't want to be analyzing traces on servers that are busy doing production work.

One consideration when stopping a trace is the file name.  You will probably want to keep the suffix the same (.ETL), but you may want to name the trace file to include the computer name, date, time, or other aspects of the trace.  This just helps later if you have multiple trace files to manage from multiple systems.

Another consideration is identifiable information in the trace.  When traces are taken, items are recorded such as computer names, file names, paths, etc.  This is just an additional consideration.  One way to get the recorded data and scrub out private information is to export to a file from the trace, and then filter or delete from your second analysis tool, such as Excel.  There are other ways to accomplish the data scrub if needed, such as PowerShell scripts, LogParser, etc.

Trace Analysis

There are two different ways to initiate trace analysis with the WPT; command line or GUI.  With the GUI tool, you can start the trace viewer by running "Xperfview.exe" from the Start menu, then from the tool called "Windows Performance Analyzer", click File, then click Open, and then browse to the trace file being analyzed.  If you prefer the command line, run "Xperf <filename.etl>".  If you run Xperf from an "administrator" level .CMD prompt, you will get the following message pop-up:

Clicking Yes will allow the trace processing to continue, clicking No will terminate the trace processing at that point.  Once the trace begins to load in the trace tool, you will see activity as the trace processing moves through it's phases:

Once the trace is fully loaded, you will see the trace depicted graphically in the Xperfview tool.  For storage performance analysis, you can get down to I/O by starting at the "Disk I/O" chart, as shown below in a sample trace:

From the screen capture above, and in the DIsk I/O chart context, you have the following elements:

• X-Axis: This is the horizontal "time" line, which is from the time the trace initiated until the trace capture stopped (in seconds)
• Y-Axis:  The value of the vertical Y-axis will change depending on this chart you are viewing.  In the first example above, you can see I/O counts.
• Top-Right "flyout": (downward pointing arrow to the right in the bar labeled "IO Counts") Changes with chart. In the top example, this is I/O Counts with red being read I/O and the blue being write I/O
  Clicking once will open the flyout, clicking again will close the flyout.
• Left middle "flyout": (right pointing arrow midway up the left side of the chart) This is the "Frame List", and allows you to select or deselect charts for immediate viewing in the Xperfview tool.
  Clicking once will open the flyout, clicking again will close the flyout.

Here is the same chart as shown above, with both flyouts open:

At this point, we will drill down into the data.  Point your mouse cursor to a section of the top-most chart, left-click and hold, and then swipe the mouse right to select an area to drill down into.  Your selection may look something like this:

Now, point the mouse cursor to any part of the shaded area of the "Disk I/O" chart, right-click, and you will have a context menu pop-up with additional actions available:

For our purposes, point to the option called "Detail Graph", and then left-click Detail Graph to open a new chart that looks like the following:

This new chart is called the "Disk I/O Detail" chart.  The first thing that stands out is the visual representation of the I/O on the disk during this trace sample.  For example, you can quickly spot trends such as sequential I/O, random I/O, or concurrent I/O with mixed workloads such as sequential and random at the same time on the same disk

The various elements of this chart are as follows:

• Detail about the disk on the left pane
• Left-upper pane drop-down, under the section called "Physical Disk".  This allows you to change the view to a different physical disk
• Upper-right side flyout titled "Process Name".  Clicking this allows you to select and deselect I/O from various process that were running at the time the trace was captured.
• X-Axis (time):  This is the time, in seconds, of the area previously selected when we swiped to select an area of the previous chart for analysis.
• Y-Axis (Offset):  This is the offset from the beginning of the physical disk, to the end of the physical disk.
  NOTE: The Offset value next to the color-chart is decimal, and the Offset value in the information pane on the left side is in hexadecimal.

Next, choose the "Physical Disk" that you would like to analyze.  in the example above, there is a "Disk 0 - Intel...".  Next to that you will see a down-arrow.  Click the down-arrow and you can click to view a different physical disk:

From here, clicking a different disk, such as "Disk 1", will change the chart to reflect the new disk selection.  You will get information about the disk, as well as the partitions on the disk.  The graphical view is a depiction of the disk and the I/O that occurred during the sample interval, with time going left to right and location of I/O on the disk going bottom to top.  In this example, the trace interval was 700 seconds, and there were 2 concurrent workloads to relatively small files, concurrently.  There was one file on a partition with drive letter "D", and another file on a second partition with drive letter "G".  For a concurrent workload like that, you would normally expect there to be some penalty involved with disk seek, as would be the case with rotational disk drives.  But this test was performed using a Solid-State Drive (SSD), so there is no seek penalty.

Now, point to an area of the chart above, somewhere towards the left side of the graph, left-click, hold, and swipe right to highlight a section of this graph to analyze further.  Your selection may look something like this:

The next image depicts Xperf with the partition information expanded (click the plus-sign next to partition to expand) and the "right-click" context menu.with "Summary Table" highlighted:

On the context menu, click the option called "Summary Table" to open a new window, which is a detail view of the I/O that took place during the interval of the sample selection.  Your chart might look like this:

This is a table, that is the result of every I/O from the sample that we drilled down into from step 1.  Recall that the first step we chose maybe all, or maybe part of that first chart.  Then in the second step we once again had a choice to select the entire chart, or only a portion.  This table contains a wealth of information about the I/O that occurred during the trace collection.

The elements of this table are as follows:

• Each line represents a single I/O, numbered based on the number of I/O in this sample interval
• Each column contains information relevant to that I/O
• You can sort the table by the column of your choice.  The sample above is sorted by "Complete Time", which will display the trace I/O top to bottom, via time of I/O completion.  You could for example click the column header "QD/I", and then you could see what was going on at various points with respect to I/O queueing.

What to Look For

In this sample we see a read I/O that took around 130 ms to complete, which is not very good.  We can see from the column QD/I that the disk queue length at this time was 508, and the inferred time spent waiting on hardware was .282 millisecond.  On an SSD drive, .282 ms I/O time is pretty common.  On a rotational drive, you would see similar response times at various points in your trace..  If you were viewing a trace from a rotational disk, and you see sub-millisecond response time, you can conclude that this I/O was serviced by cache at some point, either the cache directly on the disk, or cache on a storage controller somewhere.  What we can conclude from this I/O was the slow response time was the result of too many I/O being sent at one time, perhaps from a multi-threaded application,  You might see something like that from a stored procedure in a database application, a log checkpoint, or a simultaneous burst of I/O on an already busy disk.

At this point what you want to look for is the length of this event.  We need to determine whether this was a short-lived event that we might possibly tolerate, or was this a sustained event that resulted in poor storage performance for a length of time.  In the case above, I was running stress on a disk just to see how many I/O I could build up and you can see the result.  At some point you will cross the threshold of poor performance by adding more and more concurrent I/O.  This is one of the many great uses of this table view of all the I/O.  I can line up the I/O based on response time, and then I can just determine how many of those I/O cross the boundary to poor I/O performance.  Here is an example:

1. Start by clicking the column header called 'IO Time (us)".  This will sort the I/O by overall I/O completion time.
2. Scroll down until you reach the threshold of performance from say "great" to good".  This might be where we cross the line from under 10 ms to over 10 ms.
3. Add up the total number of I/O
4. Divide the line number of the threshold into the overall total number of I/O.  For example:

 Figure 12. Disk Detail Summary Table with I/O sorted by IO Time

In Figure 12 the areas to sort on are highlighted.  Starting at the top and going counter-clockwise, I/O Time, Line Number, Total number of reads, total number of writes are all circled.  Add reads and writes to get 3,840,592.  Divide your total IO (3,840,592) by the line number (2,063,791) and the result is 0.537 (and change).  From this we could conclude that 53.7% of our I/O in this sample completed in less than 10 ms.  Or we could conclude that 46.3% of our I/O took more than 10 ms.

We then go back to our storage performance requirements.  We might require that 90% of our I/O complete in less than 10 ms.  The conclusion from this trace sample would then be that the storage subsystem does not meet our performance requirements based on the workload that the storage subsystem experienced during this trace sample.

What are We Doing Here?

Probably the first thing people want to know when troubleshooting storage performance is what to blame.  Is it the storage?  Is it the operating system?  Is it the application?  What performance measurement tools can help determine is where to start looking.  If for example we see a high number of I/O that take more than 10 ms to complete outright, as evidenced by the "Disk Service Time" statistics, we can conclude what side of the Partition Manager (PartMgr) the problem resides.  There are still a few Windows components left after PartMgr and before we hand off to the hardware.  If we determine that I/O is building up (queuing), overall I/O completion time is high but disk service time is low, then we can look towards the operating system side for our efforts.

High Disk Service Times

There are numerous reasons why Disk Service Time numbers might be high:

• Direct-Attached Storage
• Storage subsystem is not keeping up with load
• Partition Misalignment for RAID disks
• Failing component in storage subsystem
• Storage Area Network (SAN)
• Sending too many I/O at one time
• Too many hosts per fabric port
• Too many hosts per storage port
• Failing component(s) in SAN fabric

Recommendations:

• Lighten load, or spread out load over time
• Add disk devices
• Ensure partitions are aligned (Microsoft KB 929491)
• Enable Storport tracing to confirm slow I/O below the Storport layer, which is the last layer of Microsoft software in the storage "stack" before passing the I/O to the hardware miniport driver.  Storport hands I/O to the storage hardware driver, be that Fibre Channel miniport, Storport miniport (direct-attached), or iSCSI hardware driver.  See the section below on Storport tracing.
• If SAN:
• Check statistics on fabric ports between this host and storage target.  Look for:
• BUFFER_0 errors
  Fibre Channel SAN switch ports use buffers to handle incoming and outgoing frames.  There are a limited number of buffers per switch port.  If a switch port runs out of buffers it will return a "BUFFER_0" error to sender and refuse I/O.  As soon as a buffer is freed the port will return an idle frame indicating a free buffer.  These errors are not returned up to OS, though they are visible to the host-bus adapter (HBA).  By default, the HBA will probably not report or log an associated error, because these errors are presumed to be ephemeral in nature.  There may be logging in the HBA that you can enable that will return BUFFER_0 errors.  This extended logging will add overhead somewhere.
• CRC errors
  Indicated corruption within the frame
• Enc errors
  Indicates errors at the 8b/10b encoding level
• Link Errors
• Loss of signal
• Loss of sync
• NOTE: The person troubleshooting doesn't necessarily have to understand what all the statistics means down to the specification, but just understand what is considered "normal" and what is not.  Generally it is not normal to see a lot of CRC or ENC (encode) errors.  This could indicate a hardware problem.  The problem could be a SFP connector, it could be a bad fiber-optic cable.  There have been cases where all the hardware checks out good, and the problem disappeared when changing the fiber-optic cable.  In some cases a smudge on the end of the fiber cable was the root cause.  There could also be dirt of debris on the inside of the SFP connector.  These can be tough to track down, and sometimes require replacing hardware or a hardware analyzer trace to determine root cause.

Storport Tracing (For Storport storage devices)

Storport.sys in the Windows storage port driver designed to handle IO requests for most storage devices on servers.  PCs, laptops, and other computers that use SATA drives for example, or even SSD drives, probably are using the ATA port driver (Ataport.sys).

If the storage hardware in the server being measures uses a Storport driver, and is in some way suspected of poor performance, additional Storport tracing is available to report "slow" I/O.  Storport ETW tracing has been available for a few years now.  There is more information available on this topic in the following KB article:

A hotfix is available that improves the logging capabilities of the Storport.sys driver to troubleshoot poor performance issues for the disk I/O in Windows Server 2008
https://support.microsoft.com/kb/979764

NOTE: Microsoft hotfixes are cumulative.  The tracing in hotfix 979764 is available in that fix, plus all subsequent fixes.  And, that functionality would be rolled forward to any subsequent operating systems.

This additional tracing, which we would only want to enable during a troubleshooting session, will report I/O that took longer than xx ms to complete, as measured from the Storport level within the operating system.  Storport.sys is the last layer of software in Windows that an IRP will pass through before being handed off to hardware, such as an HBA miniport driver.  These events will be reported in the Windows event log.  The ETW traces are great, but this is at a level lower than even the Xperf traces, so removes one or more layers in the storage "stack" as possible causes of I/O slowdown.

High IO Times

There are numerous reasons why IO Time numbers might be high:

• Single jobs or applications sending too many I/O in short bursts
• Concurrent jobs or applications sending too many I/O in short bursts
• "Filter" drivers stalling I/O (intentionally or unintentionally)
• Anti-virus
• Screening
• Quota
• Firewall
• Multi-Path I/O
• Windows "kernel" thrashing
• Trimming operations
• Cache operations
• File/Object contention
• Volume Shadow Copy operations

Conclusion

The Windows Performance Analyzer Toolkit is a powerful addition to your tool set for analyzing storage performance.  The uses of WPT go far beyond storage, but when it comes to storage, provide an extremely valuable tool to take a very close look at what is going on with I/O on your systems.