I looked around and couldn’t find a good C# COM interop example for the related native COM interfaces so I wrote one.   This was a good excuse for me to learn a litle bit about C# and COM interoperability.  You can find the source code here, provided under the Microsoft Permissive License.

This first cut provides a set of C# classes and related values for manipulating Windows Shell Links.  The public methods, properties and enumerations (or just publics) are C# style – I didn’t let the underlying native names leak through. 

For example, an important native enumeration is the native SHELL_LINK_DATA_FLAGS enumeration.   I’ve seen some interop examples that expose the native names of things publicly in C#.   This works, but doesn’t follow the .NET Design Guidelines for Class Library Developers (in hardback on Amazon).   Exposing the native names is a bit ugly in .NET languages, and in C# in particular. 

In the public aspects of my C# I provide the native functionally in a modern C# manner.  For example, I  provide a .NET enuneration named ShellLinkFlags which provides the values in SHELL_LINK_DATA_FLAGS.   These enumerations have C# style names like RunAsUser instead of SLDF_RUNAS_USER.

Another important attribute is that all of the publics are CLS compliant.  This enables all .NET languages to easily access these classes, not just C#.

There are some limitations to this code.  I built this so I could create shell links to EXEs – primarily console applications.    This code does not yet support the IShellLink.GetIDList() and IShellLInk.SetIDList() methiods.  So, it can only be used to link to things in the file system, and not shell objects such as printers.

The Project

This code is provided in a Visual Studio 2010 project built with C# and .NET 4.0 using the client profile.

The actual shell interop code is built in a separate class library named Eidos, which is the name of my general purpose C# library.   The namespace is Eidos.Shell.   You can use this directly, or easily incorporate it into your own code.

I’ve also included a couple of basic unit tests – you can extend these as needed.

There is also a simple program that will dump the contents of a Shell link file to the console.

The Code

The primary public class is ShellLink.  This provides a set of properties and methods that provides the functionality of IShellLink but in a C# manner.

One of the biggest differences between the C# APIs and the native COM interfaces are how Data Blocks and IShellLinkDataList are treated.    These are not exposed in the my .NET classes.  instead, they are wrapped using standard properties on the C# ShellLink class.

This works really well.  It means you can simply get and set property values just like you would with a “normal” .NET class.   The IShellLink class handles all the work of creating, writing and reading the data blocks using an internal IShellLinkDataList interface.

Download the Source

You can find the source code here, provided under the Microsoft Permissive License.

Here is screen shot of the prototype – it is pretty close to the mock up.

I started off with the Expander Control Template example on MSDN.  This wasn’t a bad start – it showed me how to change expander toggle button – nice.

But, I also needed to change the font used in the expander header.  That wasn’t shown in the example.   Ok, no problem – just use a setter in the style like this - right? 

    <Style x:Key="MainViewExpander" TargetType="Expander">
        <Setter Property="FontSize" Value="18" />
        <Setter Property="FontWeight" Value="Bold" />
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="Expander">

Bzzzztttt… wrong.   That set the font appearance in the Expander header like I wanted, but all the controls in the Expander content property inherited the font appearance.  That’s not what I wanted at all.

I needed to style the appearance of the Expander header and its contents separately.

So, i dug around in MSDN a little more.  The key information turned out to be in the documentation for the Headered Content Control class.  

This is an important class as it defines the base functionality for controls including the Expander, Group Box and TabItem controls – all very commonly used controls.

They key to getting the look I wanted is using Data Templates.   I’ve seen this used in many MSDN examples and WPF blog posts but until now, I never really understood them.

In short, data templates are like control templates but instead of defining how WPF structures the visual tree for a control, data template defines how WPF displays data. 

We’ll, you might say “Duh Richard!”  That’s what the docs say….  Ya, Duh! For sure…  I didn’t grok exactly what this meant until now: they key for me was finally understanding that the data template is bound to the data it is intended to display.

All data template intentions are bound to a property on an object instance.   You can refer to this binding by using the simple most binding which is "{Binding}".  You can see how this is used below.

The key is that all Headered Content Controls have a data template for the header content.    So, setting the font appearance for the Expander header was a simple matter of adding this setter to my style

        <Setter Property="HeaderTemplate">
            <Setter.Value>
                <DataTemplate>
                    <TextBlock 
                        Text="{Binding}" 
                        FontSize="18" 
                        FontWeight="Bold"
                        Foreground="White"
                        />
                </DataTemplate>
            </Setter.Value>
        </Setter>

This worked great!  As you can see in the screen shot at the beginning of the post, I set the text appearance to the white, 18 size bold text I wanted.   Most importantly, the appearance of the content was not affected by the header’s appearance.

Now, in one case, I do want to determine the font appearance of all the children in an Expander’s content property.   As you can seen in the screen shot, the commands are in a larger, bold font style. 

To accomplish this, I simply created a style derived from the main expander style -like this:

    <Style x:Key="MainViewExpanderCommands" 
           BasedOn="{StaticResource MainViewExpander}"  
           TargetType="Expander"
           >
        <Setter Property="FontSize" Value="20" />
        <Setter Property="FontWeight" Value="Bold" />
    </Style>

Then, in the MainView.xaml file, I used the derived style like this:

        <Expander 
            Header="Commands" 
            Grid.Column="0" 
            IsExpanded="True" 
            Style="{StaticResource 
                 MainViewExpanderCommands}"
            >
            <StackPanel>
                <Label Content="Open"    Foreground="#FFB4B4A9" />
                <Label Content="Save"    Foreground="#FFB4B4A9"/>
                <Label Content="Save As" Foreground="#FFB4B4A9"/>
                <Label Content="Close"   Foreground="#FFB4B4A9"/>
            </StackPanel>
        </Expander>

This worked fine – mostly.  If you look, you will see that each of the labels has the foreground set to a grey color.   I tried to do this in the derived style, bit alas, this didn’t work because the the intervening stack panel (which is the content for the expander) does not have the foreground property, so it is not inherited.

Here is the source code download

This code builds with the release candidate version of Visual Studio 2010.   It does not need any additional components (like NUnit).

Installing and building the project is easy:   Just download and run the installer.   It will put the project on your desktop.  Note, it doesn’t really ‘install’ anything – there are no shortcuts and it doesn’t write to the registry.  You can move the sample directory wherever you would like, or just delete it when you are done.  No un-installing is needed.   The contents are licensed with the Microsoft Public License.

Credit – I’d like to thank Josh Smith for his excellent examples of using the MVVM design pattern.   All my new WPF projects are MVVM and I learned the ropes from his blog posts.  The MVVM classes in this example are inspired by Josh’s MSDN article here.   I’m still a noob with both WPF and MVVM – the learning curve is very steep, and long.  But Josh’s blog and articles have been hugely helpful (as have many others…)

In my first example, I had a single simple animation that was either running or stopped.  The animation behavior is now like this:

I tried to do this all in XAML but it proved quite complex to handle in in that language.   This was complicated as there are two key dependency properties to deal with that leads to several states: The Spin property controls the actually spinning state.   And the IsEnabled property inherited from UIElement.  Both must work correctly together.

This gave me two significant problems:

1. The XAML was really complicated and difficult to read.  I don’t know about you, but to me, XAML is very verbose and difficult to read.  Syntax highlighting helps, but only to a degree.   There is also no such thing as a XAML debugger so a lot of trial and error is involved with large amounts of XAML.   Kaxaml is really helpful, but this was hard even with that tool.
   
2. I couldn’t get a fully correct state machine in XAML: No matter what I tried I could toggle the IsEnabled and Spin dependency properties and cause the spinner to “jump” unattractively.   
   

So, I simply used C# code to do this.   This approach was much simpler.   It was easy to implement the two cooperating state machines needed to implement all cases.   The spinning state machine is shown to the right.   It has four states and six transitions – not at all difficult to implement.

This state machine is implemented in the ControlSpinning() function on line 432 in BizzySpinner.xaml.cs.

The transitions caused by the SpinUp and SpinDown values are straight forward and implemented using a couple of switch statements

The transitions labeled Acceleration Complete and Deceleration Complete are more interesting:  they are implemented using the Completed event on the DoubleAnimation using to spin the control. 

For example when the spinner goes form the Not Spinning state to the Accelerating state, the code simply attaches a function to the animation’s Completed event like this:

This is the code that causes the spinner to accelerate to its normal spinning rate.  The key line is #7.   When the spinner has finished accelerating the SpinContinuously member is called.  It looks like this:

Note line 3 which disconnects the SpinContinuously  from the animations Completed event.   This is super important as we don’t want this called again because the next state transition when the animation completes is a different one.

The other important state machine is the one that handles the IsEnabled and Spin dependency properties.   This is what sends the SpinUp and SpinDown commands to the spinning state machine.

This is simpler than it looks:  there are only four important transitions – the ones in green where the SpinUp and SpinDown commands are sent to the spinning state machine.

This is all handled easily in the IsEnabledChanged event handler like this:

The resulting animation is very pleasing – the control spins up and down smoothly.   Its rotational rate can also be controlled dynamically.

You will need NUnit 2.5.3 (download) as I’ve added assertions. The code builds for C# 3.0 and .NET 3.5 using the client profile.  As with the last post, the project is for the Visual Studio 2010 BETA which you can download for free..

Installing and building the project is easy:   Just download and run the installer.   It will put the project on your desktop.  Note, it doesn’t really ‘install’ anything – there are no shortcuts and it doesn’t write to the registry.  You can move the sample directory wherever you would like, or just delete it when you are done.  No un-installing is needed.   Also note that the contents are licensed with the Microsoft Public License.

Here is the source code download

My graphical design skills are pretty lame, but I saw some others on the net and Mine doesn’t look too bad. 

I’m still learning WPF.  I started about two years ago (see my old posts) but was distracted by shipping a little thing we call Windows-7.  Given our team is very likely to build some production WPF based products, I need to get back on the stick and learn the WPF basics.

As with any new project, I found several good examples on the ‘tubes.  Sacha Barber’s example on CodeProject.com  was a good start and where I got the inspiration for my basic graphical design.   This one is similar, and this one is native on GDI+.  Here is another one on the Expression Behaviors Gallery.

While all are good examples, I wanted to code my own completely from scratch.   I had a few goals in mind:

1. Full support for styles and templates
2. Fully sizeable
3. Rotate in either direction
4. Could be disabled – like most other controls
5. Could be ‘spun’ manually –without an animation

This means I need to build a fully custom WPF control – a class that derives from System.Windows.Controls.Control.  This turned out to be really easy – once I got past the learning curve!

Until recently, my WPF code and XAML were pretty klunky – the code seemed complicated to me and had a faint ‘just not right’ code smell.  The more I looked at it, the riper it got.   You can see some of this stinky cheese here.   [ wow?  was that really two years ago?  I suppose that’s why they call shipping Windows-7 ‘work’ ]

In any case, I’ve had the time to do some hacking recently and I’ve learned a ton.   One thing I’ve noted is there is a LOT more WPF information on the ‘tubes than there was two years ago, and the WPF MSDN Library documentation is way, way better.   This made my recent efforts much easier.

The major thing I learned recently is that I had been going about WPF develop all wrong.  All my previous hacks had been procedural in nature – doing the UI work in code.   I hadn’t really groked how fundamental dependency properties and data binding are to WPF development.  These simply are not framework features you can use – they are fundamental constructs.  Tying things together with dependency properties and data bindings is just they way to do things in WPF.   It took me a bit to grok it, but I had one of those ah-Ha! moments are few days ago.  Groovy.

As with my other posts, I’m including the fully buildable source code.  The code currently uses the .NET 3.5 Client Profile but it builds with Beta-2 of Visual Studio 2010.     You can find the code attached.

The current code is fully functional. I’ve packaged the spinner its own assembly called Fenestra which Latin for ‘window’, clever huh?  I’ve also included a developer framework (DF) that lets me test the control.  It looks like this:

The DF lets me control the properties including the size, the spin rate and watch the events happen.   Its very simple code.  In the image above you can see two spinners, the one on the left is the default theme for the control.  The one on the right is a templated and styled version.   

All of the elements are fully style-able.   Each item in the spinner is a WPF Shape which means their key attributes can be bound to the control’s dependency properties.

I drew the fundamental shapes in Expression Design 3 which has a learning curve all its own.  I used it because it directly produces XAML.   Once I got the basic knack of using the tool, it wasn’t too hard to do.  You can find the original design files in the BizzySpinnerDF  directory.  They are named Circles.design and Rectangles.design. 

Installing and building the project is easy:   Just download and run the installer.   It will put the project on your desktop.  Note, it doesn’t really ‘install’ anything – there are no shortcuts and it doesn’t write to the registry.  You can move the sample directory wherever you would like, or just delete it when you are done.  No un-installing is needed.   Also note that the contents are licensed with the Microsoft Public License.

Here is the download

Bruce Dawson is a performance analyst on the client performance team.  He has written this guest post on enabling stack walking using xperf for both 32-bit and 64-bit Windows systems (Vista and Win7).  For more posts on xperf see this page.

Introduction

When I first started working with xperf I was confused by the many gotchas and settings surrounding the recording of call stacks. It seemed like there were many bits of crucial information needed in order to successfully record call stacks, and these bits were never gathered in one place. In order to save future generations from this complexity (and to give me a convenient reference to look at) I decided to write up what I have learned, while the lessons are still fresh in my memory.

Warning: The syntax examples below make use of ‘-‘ characters and the single quote character. Both of these characters are often altered by word processing software to visually similar but actually different characters, which can lead to obscure syntax errors. If the example syntax doesn’t work then try typing in the command manually instead of using copy/paste to eliminate that as a factor.

Recording Stack Walks

In order to enable call stacks in xperf you need to choose what type of event you want call stacks for, make sure that event is being recorded, and then enable call stacks for that type of event. The sampling profile events from the kernel provider (interrupting the CPU every millisecond and recording what it is doing) are one of the main uses of call stacks. The profile events can be enabled with the “PROFILE” kernel flag, or by using some of the kernel groups such as “Base” or “Latency” which include the PROFILE flag. For a list of all the kernel flags and kernel groups use:

xperf -providers k

In addition to enabling the sampling profiler we need to enable stack walking on that event with the -stackwalk command. We also need to make sure that the PROC_THREAD and LOADER kernel flags are enabled – so that the xperf tools can identify what modules the code addresses are in.

Putting this together, the following example command will tell xperf to record a trace with sampling profiler call stacks and save it to mytrace.etl:

xperf -on PROC_THREAD+LOADER+PROFILE -stackwalk profile

rem Your scenario goes here…

xperf -d mytrace.etl

Alternately, the following command will also work, because the “Latency” kernel group includes all of the necessary kernel flags.

xperf -on Latency -stackwalk profile

rem Your scenario goes here...

xperf -d mytrace.etl

Many events can have stack walking enabled using this basic method, so that if you want call stacks for context switches, registry operations, etc., you can get them. You can see the list of events that can have call stacks enabled for them with:

xperf -help stackwalk

You can also record call stacks for manifest-based ETW (Event Tracing for Windows) events, but the syntax is quite different, and it only works on Windows 7 and above. When you specify your ETW provider to xperf after “-on” you can specify extra parameters after the provider name, separated by colons. These are flags, a level, and, for manifest-based providers, a list of extra data to record, which can include call stacks. You can leave the flags and level fields blank and just specify ‘stack’ (in single quotes) after three colons like this:

xperf -on Latency -stackwalk profile -start browse -on Microsoft-IE:::'stack'

rem Your scenario goes here...

xperf -stop browse -stop -d mytrace.etl

The syntax above starts the kernel session with the “Latency” set of kernel flags, enables stack walking on the profile events, starts a user session called “browse”, and enables the “Microsoft-IE” ETW provider for that session, with call stacks recorded for all of those events. Then, when your scenario is finished it stops the user session, stops the kernel session, and merges the kernel and user traces into “mytrace.etl”. It’s a bit of a mouthful, but it works.

Operating System Support for XPerf Stack Walks

Recording stack walks from xperf is not supported in all variations of Windows and does have some prerequisites. Note also that stack walking for classic ETW events is not supported. Owners of classic ETW providers must upgrade their providers to manifest-based to take advantage of stack walking on their custom events.

The following chart summarizes the situation for stack walking on kernel and manifest-based events:

* Stack walking not supported for manifest-based ETW events.

** Stack walking on 64-bit requires that the DisablePagingExecutive registry key be set.

Disable Paging Executive

In order for tracing to work on 64-bit Windows you need to set the DisablePagingExecutive registry key. This tells the operating system not to page kernel mode drivers and system code to disk, which is a prerequisite for getting 64-bit call stacks using xperf, because 64-bit stack walking depends on metadata in the executable images, and in some situations the xperf stack walk code is not allowed to touch paged out pages. Running the following command from an elevated command prompt will set this registry key for you.

REG ADD "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG_DWORD -f

After setting this registry key you will need to reboot your system before you can record call stacks. Having this flag set means that the Windows kernel locks more pages into RAM, so this will probably consume about 10 MB of additional physical memory.

Looking at Stack Walks

The call stacks that result from the stack walks are visible in xperfview. You can run xperfview and load your trace or you can invoke it from the command line with:

xperf mytrace.etl

If you enabled the PROFILE kernel flag then you should see one or more CPU Sampling graphs – if they aren’t enabled then use the selector on the left sidebar, or the Graphs menu, to enable one of them.

From one of the CPU sampling graphs you can select an area of interest and then right-click and select Summary Table. Use the selector on the left sidebar of the summary table, or the Columns menu, to make sure that the Stack column is enabled and visible – this is where your call stacks will be shown. Xperfview summarizes multiple stack walks together, collapsing them down where they are identical. The Count field shows how many stacks were collapsed together on each row (how many times that partial call stack was hit), the Weight column shows an estimate of how many milliseconds of CPU time were spent in that call stack, and the %Weight column shows that as a percentage of total CPU time available.

Sampling Implies Estimation

CPU sampling is a statistical profiling process. If the number of samples is small, or if the code execution is correlated with the sampling interval, then the results can be off by a large amount. It is important to be aware of the limits of sampling in order to avoid reading too much meaning into a small number of counts.

Symbols

If you haven’t loaded symbols then your call stacks will show DLL names followed by question marks, as shown below. You can tell that you’ve recorded a call stack, but the details of what code was executing are still hidden.

A single question mark in the stack column means that no call stack is available at all. If “?!?” is displayed then that means that xperf doesn’t know what executable image was at the address.

If the DLLs are from Microsoft then you should point your symbol path to Microsoft’s symbol servers (using the _NT_SYMBOL_PATH environment variable or Configure Symbol Paths from the Trace menu). You can have multiple symbol paths, separate by semicolons, to that you can point at Microsoft’s symbols and your own simultaneously. Then you need to select Load Symbols from the Trace menu or a graph context menu.

Exploring Call Stacks

The call stack display is a bit unusual in that the root of the call stack (conveniently labeled as [Root]) is displayed at the top and you expand downwards from there. The layout is also a bit subtle and non-obvious at first. When you see multiple functions above each other at the same indentation with no minus signs in front of them (for example endthreadex and _AfxThreadEntry below) then that means that in that section of the call stack all of the collapsed stacks took the same path – endthreadex called _AfxThreadEntry in all of the samples and these functions have a parent/child relationship. This layout helps to keep the call stack display as compact as possible.

When you see multiple functions above each other at the same indentation with minus signs in front of them (for instance Calculator::Calculate and IterDataImp<int>::Render) it means that all of these functions are called by the function above them. In other words those functions are siblings, all called by HomeWork::CalculateNextBit, and the collapsed call stacks took multiple paths at this point.

When you see a line that ends with <itself> then than line represents samples that occurred in that function, as opposed to in its descendants. This is often described as the exclusive time, as opposed to inclusive time which includes descendants. The <itself> line for a function, if present, will show up as a child node of the function so that the child nodes’ counts sum up to the parent’s count. In the example above Calculator::Calculate was running for 2743 samples and its child function was running for 2156 samples, for a total of 4899.

The best way to get the feel for how this works is to play around with it. If you select a call stack and repeatedly press the right arrow key then the call stack will keep expanding down the hottest path and this can be a very effective way to drill down into what is probably the most important call stack.

To see call stacks for manifest-based ETW events (assuming you enabled them) you need to go to the Generic Events graph and then select Summary Table from its context menu. From there you should make sure that the Stack column is enabled and you can then explore the stacks. The Count column can be very useful to see how many times the various events were hit from various call stacks. When enabled the stacks of manifest-based events are also available in the tooltips of the corresponding event markers in the Generic graph itself.

Summary

Call stacks can be recorded with:

xperf -on Latency -stackwalk profile

rem Your scenario goes here...

xperf -d mytrace.etl

If you are tracing on 64-bit Windows don’t forget to set DisablePagingExecutive and then reboot before recording call stacks. And remember that Windows 7 has better ETW call stack recording support than Windows Vista, and Windows XP has no ETW call stack recording support.

To analyze call stacks load the trace into xperfview and look in the appropriate summary table in the Stack column.

Happy tracing!

Bruce Dawson, Windows Client Performance Team

The program has a very simple, but fully functional UI.

The “Open a file…” pops up the standard open file dialog and ask the user to find a file.   The cancel button is disabled unless there is a loading operation to cancel.  The text box is for messages from the loader thread and the slider is just there to show that the UI is responsive while the background load occurs.

The progress bar is hidden unless a file is actually being loaded.  If the async check box is un-checked, then the file will be loaded synchronously.  This illustrates how the UI will ‘freeze’ if a second thread isn’t used.

The key thing about this program is that the UI stays completely responsive while a file is loaded.  Of course, this can be done without threads – many programs are written using a state machine approach.  But using threads, and functional programming make this much more simple than previous approaches.

There are many ways to leverage threads in managed code, but the most straight forward way is using the BackgroundWorker class.  IT can be used in two ways, by creating one and attaching functions to its delegates, or by creating a new class derived from BackgroundWorker – this is the approach I took here.  

I created a class called AsyncFileLoader that is derived from BackgroundWorker.  We’re using all the features of BackgroundWorker here

1. Support for asynchronous progress notifications
2. Support for cancelation
3. Handling of errors during the file loading

Progress and cancelation is enabled by setting their properties in the constructor.

   1: public AsyncFileLoader()

   2: {

   3:     WorkerReportsProgress = true;

   4:     WorkerSupportsCancellation = true;

   5: }

We also need an event handler that will be connected to the cancel button

   1: public void PerformCancel( object sender,  RoutedEventArgs e )

   2: {

   3:     this.CancelAsync();

   4: }

The final element is a function that overrides BackgroundWorker.OnDoWork().   This function does all the work of asynchronously loading a text file.

   1: protected override void OnDoWork( DoWorkEventArgs e )

   2: {

   3:     // note: do not catch exceptions here.  The BackgroundWorker will catch exceptions 

   4:     // and pass them back to the initiating thread in the RunWorkerCompleted() delegate 

   5:     // using the error member of RunWorkerCompletedEventArgs parameter.

   6:  

   7:     string FileName = e.Argument as string;

   8:  

   9:     ReportProgress( int.MinValue, String.Format( "Opening {0}", FileName ) );

  10:  

  11:     using ( StreamReader sr = new StreamReader( FileName, Encoding.UTF8 ) ) 

  12:     {

  13:         ReportProgress( int.MinValue, String.Format( "Loading {0}", FileName ) );

  14:  

  15:         long FileLength = sr.BaseStream.Length;

  16:         long PercentRead = 0;

  17:         long LastPercentRead = 0;

  18:  

  19:         String line;

  20:  

  21:         while ( (line = sr.ReadLine()) != null ) {

  22:  

  23:             if ( CancellationPending ) {

  24:                 e.Cancel = true;

  25:                 break;

  26:             }

  27:  

  28:             PercentRead = (sr.BaseStream.Position * 100) / FileLength;

  29:  

  30:             if ( PercentRead >= LastPercentRead + 1 ) {

  31:                 ReportProgress( (int)PercentRead );

  32:                 LastPercentRead = PercentRead;

  33:             }

  34:         }

  35:     }

  36: }

As you can see – this is really simple, and most important procedural – there isn’t a state machine, or any explicitly state kept at all.  This single function loads the file, and sends progress messages and errors asynchronously to the calling thread.

The initiating thread passes the background thread parameters using the DoWorkEventArgs class.  Here, I’ve simply passed the file name as a string, but any class or structure could be passed this way.

Line 9 is important – it asynchronously passes the first progress message back to the main thread.   This includes the progress percentage and the name of the file.  Here I’m using int.MinValue to differentiate the first progress message from subsequent ones.  This sentinel value means that the main thread doesn’t have to maintain any state to handle progress messages.

I mentioned above that this function handles errors, but you don’t see any error handling code!  Any errors that occur from opening the file, or reading the data are signaled as exceptions.  The BackgroundWorker class will catch these and pass them back tot he main thread.  I just love this – it keeps the code here very clean.

Note the using statement on line 11.  It is important to do this when using file I/O objects so the underlying OS objects get cleaned up correctly.  See this for more information.

The progress percentage value is computed very simply, the code simply keeps track of the file position as a ration to the file size.  This is the obvious part.

Note line 30 – this is very important.  It only sends progress messages to the main thread when the value changes – not for every line, which would be quite expensive (give it a try to see).

The entire AsyncFileLoader.cs file is not very big, only 29 lines of actual code and is very straightforward.

The code in the main thread to mange the UI is equally straight forward and consists of just a single event handler attached to the open button’s click delegate.  This one function does all of the following:

1. Gets the file name from the user using the open file dialog.
2. Maintains the UI state, disable buttons,  showing/hiding the progress bar, displaying messages, etc.
3. Creates a the AsyncFileLoader object
4. Starts the asynchronous file loading operation
5. Handles progress messages
6. handles the completion of the operation
7. Handles errors.

   1: private void OpenAFile_Clicked( object sender, RoutedEventArgs e )

   2: {

   3:     try {

   4:         OpenFileDialog ofd = new OpenFileDialog();

   5:         ofd.CheckFileExists = true;

   6:         ofd.DefaultExt = "txt";

   7:         ofd.Title = "Open an text File";

   8:         ofd.Filter = "Text|*.txt";

   9:  

  10:         OpenAFile_Button.IsEnabled = false;

  11:  

  12:         ofd.ShowDialog();

  13:  

  14:         if ( ofd.FileName.IsBlank() ) {

  15:             OpenAFile_Button.IsEnabled = true;

  16:             return;

  17:         }

  18:  

  19:         AsyncFileLoader TheLoader = new AsyncFileLoader();

  20:  

  21:         //---- Do this when the progress bar is updated

  22:         TheLoader.ProgressChanged += ( Sender, ArgsX ) => {

  23:  

  24:             if ( ArgsX.ProgressPercentage == int.MinValue ) {

  25:                 Cancel_Button.IsEnabled = true;

  26:                 Cancel_Button.Click += TheLoader.PerformCancel;

  27:                 OpenProgress_ProgressBar.IsEnabled = true;

  28:                 OpenProgress_ProgressBar.Visibility = Visibility.Visible;

  29:                 OpenProgress_ProgressBar.Value = 0;

  30:                 Info_TextBox.Text = ArgsX.UserState as String;

  31:             }

  32:             else {

  33:                 OpenProgress_ProgressBar.Value = ArgsX.ProgressPercentage;

  34:             }

  35:         };

  36:  

  37:         //---- Do this when the background worker is completed 

  38:         TheLoader.RunWorkerCompleted += ( SenderX, ArgsX ) => {

  39:  

  40:             RunWorkerCompletedEventArgs args = ArgsX as RunWorkerCompletedEventArgs;

  41:  

  42:             if ( args.Error != null ) {

  43:                 PopErrorDialog( args.Error, "Cannot process file!" );

  44:             }

  45:             else

  46:             if ( args.Cancelled ) {

  47:                 Info_TextBox.Text = "Loading cancled";

  48:             }

  49:             else {

  50:                 Info_TextBox.Text = "File Loaded";

  51:             }

  52:  

  53:             Cancel_Button.IsEnabled = false;

  54:             Cancel_Button.Click -= TheLoader.PerformCancel;

  55:             OpenProgress_ProgressBar.IsEnabled = false;

  56:             OpenProgress_ProgressBar.Visibility = Visibility.Hidden;

  57:             OpenAFile_Button.IsEnabled = true;

  58:         };

  59:  

  60:         //---- Start the ascynrnous file loading operation

  61:         TheLoader.RunWorkerAsync( ofd.FileName );

  62:     }

  63:     catch ( Exception Ex ) {

  64:         PopErrorDialog( Ex, "General Failure!" );

  65:     }

  66: }

You can see in the progress changed handler how no state needs to be kept – this lambda function gets all the data it needs from the event.   You can see how the sentinel value (int.MinValue) is used to trigger the change in UI state.

Completion is handles in a similarly manner and resets the UI state.   This lambda also handles any errors thrown by the loader by checking the args.Error member which will reference an exception if one was thrown.

This is a great example of using multi-threaded programming for something other than speeding up a computationally bound algorithm using parallelization.   Some may suggest that this isn’t “real multi-core” programming, but it is.  Indeed, the developer doesn’t need to know how to implement complex inter thread communication or synchronization, but just because this isn’t hard, doesn’t mean it isn’t real multi-threaded, and multi-core enabled code.  It absolutely is and the developer does need to understand asynchronous programming techniques to use it.

Using the BackgroundWorker class is a very simple straight forward way to enable your programs to be more responsive, and to leverage multi-threaded and multi-core performance.   You can use other managed code techniques such as threads and dispatching for more sophisticated work.  

You can find the complete VS 2008 project in the attached ZIP file.

BackgroundFileOpenExample.zip

Recently, I ran across this question “Are you concerned about multicore”.   HenryR, a PhD candidate at Cambridge is asking if the “developer on the street” needs to concern him/herself with multi-core development  practices.  

Henry’s question has a few answers, including the accepted one dmckee (a particle physicist)which I’ll focus on here.

There are many types of programs such as

• A simple script one of might write to do some work then throw away,
• high performance very parallel scientific  applications,
• Main stream operating systems
• Line of business applications
• Web based apps
• Command line based applications
• Applications with graphical user interfaces.

This is simply the tip of the ice berg – I’m confident that with a little brainstorming, this list would be come large.

Dmckee’s answer is correct – but only for a small set of things.  Indeed, for a very simple application, that is not CPU bound, then making it multi-threaded may be more work than needed.  For example, I have utilities that are not multi-threaded.

However, even utilities can take advantage of multi-threading, and I argue that its de rigueur for a production program to do so.

The astute reader may say “But wait a minute!   Henry ask about multi-core, not multi-threading…”.

Yup, but to leverage multi-core, you need to write multi-threaded programs.   Using more than one thread has other advantages beyond parallelizing CPU bound operations so they have better throughput.

To pre-fetch the next comment “But if its not CPU bound, then why bother with multi-threading?  That’s just a waste of time!”

In my mind, there are three reasons for this:

1. It can make your program more responsive from a user perspective.
2. Even for things that are not CPU bound, it makes sense to do them in parallel.
3. It can break up complex single threaded state machines into simpler, more procedural code.

In Windows, the user interface is message based: Windows sends a UI program a message for ech UI event the program needs to handle.   A program with any kind of graphical UI uses the GetNextMessage() family of functions to get them.  I don’t know about Linux or OSX, but I suspect they are similar in this regard.

This is conventionally done in a message loop where a program loops on getting then processing messages.   If there are no messages, then the program can wait (block) for a new message, or do some background, or idle processing.

Messages included all kinds of things, but most importantly user input – mouse moves and clicks, and keyboard events.  These are often translated into windowing operations (moves, resizes, etc) and control actions (button clicks, scrolling, etc).  In short, this means that coding handling the UI is a big state machine and that this state is not kept on the stack.  It has to be kept explicitly somewhere else.   

This has an important implication – for an application to main responsive, it must be able to quickly and consistently service the message loop.   This is done on a single thread, indeed many applications only have a single thread that does everything. 

Windows programs have been written this way for a long time.  Some people might argue that this is a design deficiency.  That’s a topic for a different discussion about application compatibility – do remember, this pattern was set back in the very early 90’s – Windows 3.11 was released in 1992.

In any case, delays in the message loop that the user can perceive are called hangs.  There are two ways to keep the message loop from hanging.

1. Break up all operations into pieces that execute quickly enough so that the message loop can always be serviced quickly.   Indeed, lots of developers have tackled the problem just this way.
   
2. Move longer running operations to another thread, or threads.

The short story is that only #2 works effectively. Why?  Simply put – I/O.  Specifically disk and network I/O.  But other I/O, such as GPU operations can also be an issue.

The problem is that any disk or network I/O operation can potentially be long – hundreds or thousands of milliseconds long.  Programs that do their file I/O and networking I/O on the UI thread will have hangs.  It is unavoidable.

Some may point out that “Hey!  That doesn’t matter, making it multi-threaded doesn’t speed things up!”.

Right…. and wrong…

Right in the sense that it doesn’t increase an applications throughput – the work still has to get done.   But wrong in the sense that it absolutely positively benefits the user.

So for point #1 multi-threading can make your program more responsive from a user perspective because you can move potentially long running operations off the UI thread – keeping the UI “alive” and responsive.   Trust me, your users will love you for this.

In my next post, I”ll discusses the source code for a C# program that shows how simple it is to move operations off the UI thread in WPF applications.   This doesn’t require any complex knowledge of fine grained synchronization, or complex multi-threaded programming.  The code is also easy to understand, not state machine driven and easy to debug.   All things that conventional wisdom holds is true for multi-threaded programming.

In future posts, I’ll talk more about the example source above, points #2 and #3 , and some general multi-core and multi-threaded topics.

In summary – multi-core and multi-threaded programming is much more than simply speeding up (parallelizing) compute bound operations.  Efficient threading can bring other benefits to your programs as well. 

First, its important to understand that a trace file (.ETL) is simply just the buffers produced by trace session written to a file.  The data in an ETL file isn't pre-processed, summarized, or otherwise annotated with meta data as it comes out of the OS.  Its is just the raw data that comes from a ETW session.  This is because ETW is designed for log time efficiency - ETW does the absolutely minimal amount of work needed to get the trace data to a file, or other consumer.

This means that all the heavy lifting of post processing trace data happens later. With the xperf tools, there are two places where this occurs:

1. In the merge step, xperf takes the kernel trace and trace files and merges them into a single trace file.  Xperf will merges (adds) meta data to the trace (I've got another post that provides all the detailed on merging in the works...).  The result of merging is a single trace file that can be analyzed by the tools directly on the target machine, or copied to another system for analysis.   Note that the merge step must happen on the system where the trace was taken (the target system).
   
2. When a trace is processed xperf using actions, or loaded into Performance Analyzer, the core trace processing components do a lot of work on the raw trace data.  This includes things like mapping process IDs (PIDs) to file and process names, mapping addresses to filenames, loading symbols for address, unifying stacks, and handling 64-bit and 32-bit differences.

As you've seen in the other posts, once a trace is merged it can be viewed in the Performance Analyzer.  But, xperf also allows you to see what is in the trace using the dumper action.  This is easy to do:

xperf -i fs.etl -a dmper >fs.csv

The -i fs.etl specified that the input file is FS.ETL.    The -a dumper parameter tells xperf to execute the dumper action.   The output goes to the standard output.

There is a short cut for this as well: the dumper action is the default action so if you only specify an input file then xperf simply dumps it.   For example, the following command does the same thing as the one above:

xperf -i fs.etl  >fs.csv

The resulting file is an ANSI text file where each line is one record.   Each record consists of a comma delimited set of fields.  The first field of each line is the name (or type) of the record. 

There are some special lines and sections at the front of the file.   Each record type is described by a header line.   The header lines are delimited by the 'BeginHeader' and 'EndHeader' lines.   Note that the line immediately after the 'EndHeader' line is unique, it doesn't have a header line.  This line describes some of the characteristics of the trace such as its duration, and the pointer size.

The first field of each header line is the name (or type) of the ETW record that the header line describes.  The rest of the fields are the names of each of the fields for the record type.  Here is an example of the process start event header (P-Start) and a P-Start event.

P-Start,  TimeStamp, Process Name ( PID),  ParentPID,  SessionID,  UniqueKey, UserSid, Command Line

P-Start, 1017280, fs.exe (3004), 3608, 1, 0x86661508,
S-1-5-21-626881126-397955417-188441333-3225678, c:\coding\fs\Release\fs\fs.exe  blflargorg c:\coding\*.cpp *.h -s

This event describes the start of a process.  There is also a corresponding P-End event. For processes that are already running when he trace is begun, the kernel logger includes a pseudo P-Start event. 

This means that every PID seen in other events will have a corresponding P-Start event in the trace before it is seen in an event.

Also note that xperf will dump events that you add to your own applications so long as you include an event manifest in your app.  So, you can add your own events and use xperf to dump them in the context of all the other events you include in the trace.

Xperf, a new tool in the Windows SDK
The SDK team just shipped the latest version of the Windows SDK which supports Windows Server 2008 and Vista SP1.  The SDK now includes an important new tool; the Windows Performance Tool Kit from the Windows performance team (we call them the xperf tools for short...)

Xperf Tools Landing Page and Update
The WHDC folks now have web page setup for the Windows Performance Toolkit (aka the 'xperf tools').  The page includes downloads for updates to the versions that ship in the SDK.  In the near future, this page will include pointers to updated documentation, and discussion forums.

Using Xperf to take a Trace (updated)
Lets get to it!  Here is how to take a basic trace then look at CPU and disk utilization.    Its really simple, just three commands to turn on tracing, turn it off, and then view the trace.

Xperf support for XP
"Do the xperf tools support XP or Windows Server 2003?" is a frequently ask question.  The answer is no mostly, and yes for a few things.

Using the Windows Sample Profiler with Xperf
Using the xperf tools, ETW, and the kernel sample profile interrupt all together provides a very effective and easy to use sample profiler for the analysis of both application and system wide performance.  At each sample interrupt, the ETW sub-system captures the instruction pointer and the stack.  This data is lazily and efficiently logged to an ETL file.  Once the data is saved, it can be analyzed with Performance Analyzer.

So just what is in a trace? Using the xperf trace dumper
There is a lot of information in a typical kernel trace.  While the Performance Analyzer tool is quite powerful and makes it easy to view a trace graphically, sometimes you just need to see what is in the trace directly.  Xperf makes this easy.

Stack Walking with Xperf
Bruce Dawson is a performance analyst on the client performance team.  He has written this guest post on enabling stack walking using xperf for both 32-bit and 64-bit Windows systems (Vista and Win7).

The next article in this series is  So just what is in a trace? Using the xperf trace dumper

Note: the examples in this post only works on Vista or Server 2008 32-bit;  Prior operating system's do not support taking stack traces.  Taking stack traces on 64-bit platforms will be the topic of another post.

Here is an example of profiling FS.EXE, a grep-like utility I've written.  I use this tool for experimenting with various topics such as efficient I/O, well performing string matching algorithms, and instrumenting applications with ETW.

For this test, I put the following commands in a CMD file:

• xperf -on PROC_THREAD+LOADER+INTERRUPT+DPC+PROFILE
  -stackwalk profile
  -minbuffers 16 -maxbuffers 1024 -flushtimer 0
  -f e:\tmp.etl
  
• fs.exe farglenorgin c:\coding\*.cpp *.h -s
  
• xperf -d profile.etl

Since the commands are a bit long, I've separated them above and added line breaks to make them readable.  Each command above should be on one line in your command file.

The first command turns on the kernel logger and enables the following events:

• PROC_THREAD flag enables the process and thread events. These mark the beginning and ending of each process and thread.  The kernel provider guarantees that there will be a begin/end pair for every process and thread during the trace.   Process and threads that exist before the trace was started or are still running when the trace is stopped also have these events.
• The LOADER flag enables the loader events that log when the kernel loads an image (an EXE or DLL)
• The INTERRUPT  and DPC flags enable the ETW interrupt and DPC events which mark each interrupt and deferred procedure call which are routines that run at DISPATCH_LEVELt
• The PROFILE flag does two things; it turns on the systems sample profile interrupt and it enables the kernel's sample profile ETW event.

The other flags are important as well.

• The -stackwalk profile parameter turns on ETW's stack walking feature for the sample profile event.   Every time a sample profile event is triggered by the sample profile interrupt, ETW will capture the stack and save the data in the trace buffers.  
• The -minbuffers 16 parameter sets the minimum number of buffers that ETW will allocate for storing events.  Note, you need at least two for each processor in you system.
• The -maxbuffers 1024 parameter sets the maximum number of buffers ETW will allocate to 1024 - a total of 64MB.
• The -flushtimer 0 parameter tells ETW to never flush the buffers based on  timer, buffer's will only be written to disk when they are full.
• The -f e:\tmp.etl parameter tells ETW to lazily write the full ETW buffers to e:\tmp.etl.   This puts the log file on a different physical drive than the drive on which the experiment is running.  This means that the writes that ETW uses to save the trace data do not occur on the interesting drive.

The second command simply runs the experiment.  It searches for the string 'farglenorgin' in all my .CPP and .H files.  I'm using a string that doesn't exist so I execute the worst case code paths in the application.  Replace this command with a command to run your experiment, or a pause instruction so you can dork around with a graphical program.

The third command simply stops the kernel logger, merges the data and saves it in profile.etl.

NOTE: These commands need to be run from an elevated command prompt.   Controlling ETW tracing requires administrative privileges.

There is now one other thing to do before examining the data - setting the symbol path.   Here is how I set the symbol path for this example: 

set _NT_SYMBOL_PATH =
c:\coding\fs\release\fs;
SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

This tells the symbol decoder to look for symbols in the release build directory for FS.EXE and in the Windows public symbol server, caching the served symbols in c:\symbols.  The xperf tools uses the symbol decoding libraries from the debugging tools for Windows.  You can find more information on using symbols here.

Once the trace is taken and the symbol path is set, then simply open the trace in Performance Analyzer with the command "xperf profile.etl". 

The CPU Sampling by Process graph is the most interesting graph for this example.  To select the visible graphs, click on the flyout control on the left of the window, then select the CPU Sampling by CPU, and by Process graphs.

For his experiment, the CPU sampling by Process graph looks like this:

By default, all processes running during the trace are shown except the idle task (as seen above).  You can change which processes are displayed or hidden by by using the check boxes in the legend drop down as shown above.

This graph illustrates an important concept about the kernel event provider and the xperf tools in general - they are specifically designed to analyze system wide and application performance data and events.  For example, in the legend above, there are many processes listed, but only a few of them actually used any CPU time during the experiment.

Using the legend, you can eliminate all processes except the interesting one.   Here is what the graph of CPU utilization for only FS.EXE looks like

This is pretty cool as it provides a nice overview of the CPU utilization of FS.EXE, but it really doesn't tell us much about where time is being spent in the process it self.

The real power in Performance Analyzer is in its summary tables.  These are tabular displays of data about a specific chart, or a region in a chart.  For this experiment, I looked at the sample profile data for the entire trace.  To do this, right mouse click on the CPU Sampling by Process chart, make sure that the load symbols option is set, then select the Summary table view.

Note, it will take 10 to 20 seconds for the summary table to show up.  Performance Analyzer is loading symbols while this is happening.  (putting symbol loading on a background thread is on our to do list...).

After the summary table pops up, click on the flyout and select the columns for display.  In this case you will want the process, stack and % Weight columns (feel free to experiment with other columns). 

Next, arrange the columns as follows.  The columns to the left of the gold column are grouping columns.   You can change the order of columns and put them to the left or right of the gold column by dragging.

Now, you can expand the stacks for FS.EXE and see where it is spending its time.  Not that this isn't by function as in some profilers but by call stack.  This is much more powerful than simply knowing the functions where time is spent as it also shows you how the time consuming functions were called.

Its no surprise that my find string utility spends most of its time in the following stack:

As with other sample profilers, you can look "up" and "down" the stacks from any particular point. This is commonly called a butterfly view.  Right mouse click on any item in the stack column and experiment with the callers/callees and inntermost/outermost options, like this:

This stack trace has very simple call stacks so it isn't very useful for looking at butterfly views.  But try one of your own programs and look at a butterfly stack view of a function that is called often from multiple places.   Or, use the butterfly view too look at a intermediate function and see all the functions it calls, and their stacks.

The above screen shots and summary table views contain the data from the entire trace.   This works ok for short traces.  But for longer traces, or even short traces with a lot of detail, we often need to look at specific time spans. 

For example, there are some time spans in my experiment where FS isn't using very little CPU time.  I'd like to see what FS is up to in that time span.  This is easily done by using the left mouse button to select a time span on the X axis and zooming the graph to that view, or look at the summary table for that span, as in this example.

Once the interesting region is selected, I simply use the right mouse button to pop up the context menu and select summary table. 

Note that you can open up multiple summary tables, each from different regions of a graph, or even different graphs.  This is great for making comparisons.

The new summary table window now only shows the data for selected time span in the trace.  Is not surprising that FS is spending the little CPU time it is using in user mode asynchronous procedure calls.

This post illustrates some key concepts:

• The xperf tools are designed for both system wide and application specific analysis.
• Profiling with ETW is very, very light weight.  While the experiment is running, the xperf tools are not even loaded - the kernel itself is collecting the data.   All analysis is done as post processing tasks.
• OS based sample profiling collects both user and kernel mode stacks.  
• So long as you have symbols, production code can be profiled - no special debug or instrumented builds are required.
• In this example, I started and stopped FS.exe (the experiment) between the tracing start and stop.  But, since this is ETW based, sample profiling can be started and stopped at any time, without stopping or restating even a single process.  You can profile anything at any time on any system.
• Stack views provide a very powerful method for analyzing where time is spent in a process.
• The general technique with Performance Analyzer is to use the graphics to identify interesting time spans in the trace, then use the summary tables to look at the data in detail.