MSBuild 4.0 has all sorts of features for targeting different .NET Framework versions. The idea is that you can use MSBuild 4.0 to build all your legacy (pre-4.0) project types, as well as new projects that just target a downlevel version of the .NET Framework. In addition, you can mix and match the target framework versions for different projects under a single invocation of MSBuild.exe. Unfortunately, in Beta 2, we've had reports of a problem with building unit test projects that contain the special generated accessor assemblies, otherwise known as "test references". If you have a VS 2008 unit test project with test references, you may find that MSBuild 4.0 gives you an error like:

Caught a BadImageFormatException saying "Could not load file or assembly 'C:\Build\Project\Sources\ProjectName\ProjectName.UnitTests\obj\Debug\ProjectName_Accessor.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.".

Update: This should be fixed for the final release of VSTS 2010.

The Problem

MSBuild 4.0 uses the .NET 4.0 CLR. All the MSBuild tasks, including third party and custom tasks, are loaded under the 4.0 runtime. This, in most cases, isn’t bad, because the 4.0 runtime should be perfectly capable of running code that was targeted for the 2.0 runtime. However, in the case of the VS 2008 Team Test BuildShadowTask, the code creates an assembly using the System.Reflection.Emit namespace. That namespace, when provided by the 2.0 CLR emits assemblies targeting the 2.0 CLR. When the namespace is provided by the 4.0 CLR it emits assemblies targeted at the … drum roll please … 4.0 CLR! So, when you build a VS 2008 unit test project (targeting the 3.5 framework, which uses the 2.0 CLR) using MSBuild 4.0, you end up with accessor assemblies that target the 4.0 CLR. Oops. Now downlevel applications trying load the assembly will blow up with the above exception. You’ll see it sometimes during your build with ResGen.exe, or when you go to run your tests from MSTest.exe.

The Solution Workaround

To get around this, all you need to do is run the BuildShadowTask from MSBuild 3.5, which is the version VS 2008 projects were meant to be built with. However, if you are using build features in TFS 2010 Beta 2, you are in trouble, since it (out-of-the-box) only supports MSBuild 4.0. Sure, you could write your own build activity to run MSBuild 3.5, but MSBuild 4.0 is *supposed* to work. So rather than blaming the TFS Build team (of which I am a part of) I’ll address the real issue. Several teams have gained traction to resolve this bug before shipping, but there is no guarantee. Quite frankly, this bug is minor in regards to the entire project. My team is pushing heavily for it to get done, but time will only tell.

Just as an extra note, the problem is actually in the VS 2008 Microsoft.TeamTest.targets file, or the BuildShadowTask that it uses (depending on how you look at it). They generate an assembly targeting the wrong framework version. So that is a natural place to fix the issue. Instead of modifying the .targets file directly (you could, but think of all the things that could go wrong!) I decided to add a new .targets file, and then modify my unit test projects to load it instead of the original. The new .targets will replace the ResolveTestReferences target that Microsoft.TeamTest.targets exposes.

Fast Track

If you just want to get it done and don’t care to know how it works, do this:

1. Download the attached file and place the .targets file in %ProgramFiles%\MSBuild\Microsoft\VisualStudio\9.0\TeamTest (or %ProgramFiles(x86)%\MSBuild\Microsoft\VisualStudio\9.0\TeamTest on an x64 system).
2. Add the following property to your unit test project (.csproj or .vbproj) files near the beginning (it needs to be before the import of Microsoft.CSharp.targets).
   
   <MsTestToolsTargets>$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v9.0\TeamTest\Microsoft.TeamTest.4.targets</MsTestToolsTargets>

The Details

I am aiming to replace the old behavior. Microsoft.Common.targets uses the MsTestToolsTargets property to define where to load the test tools .targets file from, so I have an easy way to replace the old behavior. Let’s first look at the existing Microsoft.TeamTest.targets file.

  <UsingTask TaskName="BuildShadowTask" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\TeamTest\Microsoft.VisualStudio.TestTools.BuildShadowsTask.dll"/>

If you are familiar with MSBuild project files, this is easy to understand. If not, just know that the UsingTask element is a way to import a coded MSBuild task. In this case, the BuildShadowTask being imported is what generates the accessor assembly. Well, sort of. It links to Publicize.exe, which does the actual assembly generation, but as far as MSBuild is concerned, the task’s implementation is irrelevant.

  <PropertyGroup>

    <ResolveReferencesDependsOn>

      $(ResolveReferencesDependsOn);

      ResolveTestReferences

    </ResolveReferencesDependsOn>
  </PropertyGroup>

In this section, the ResolveReferencesDependsOn property, which is used to determine the targets that the ResolveReferences target depends on, is modified to include the ResolveTestReferences target. The ResolveTestReferences target is the target that invokes BuildShadowTask, which you’ll see below. This is a standard way to add your custom targets to the common target dependency chain.

  <Target Name="ResolveTestReferences" Condition="'@(Shadow)'!=''">

    <BuildShadowTask

        CurrentResolvedReferences="@(ReferencePath)"

        CurrentCopyLocalFiles="@(ReferenceCopyLocalPaths)"

        Shadows="@(Shadow)"

        ProjectPath="$(ProjectDir)"

        IntermediatePath="$(IntermediateOutputPath)"

        SignAssembly="$(SignAssembly)"

        KeyFile="$(AssemblyOriginatorKeyFile)"

        DelaySign="$(DelaySign)">

      <Output TaskParameter="FixedUpReferences" ItemName="ReferencePath"/>

      <Output TaskParameter="NewCopyLocalAssemblies" ItemName="ReferenceCopyLocalPaths"/>

    </BuildShadowTask>
  </Target>

Here is the meat of the .targets file. It is a single target that leaves the bulk of the logic to the custom task. It passes all the relevant properties and items to the task, and assigns a couple of outputs. What you need to take from this is that when I re-implement this target, I need to maintain the same behavior. I looked at invoking Publicize.exe to accomplish this, but it turns out that Publicize.exe doesn’t have very many command line parameters. It could be done, but you’d need a custom task or a lot of MSBuild logic to accomplish the same as the standard BuildShadowTask. So, instead, I cheated. My solution was to shell out to MSBuild 3.5 to do this part of the build process. That means I need to be able to target the ResolveTestReferences target alone, which took a little extra work.

Let’s jump into the new .targets file. Remember, this replaces the above logic, it doesn’t add onto it.

  <UsingTask TaskName="BuildShadowTask" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\TeamTest\Microsoft.VisualStudio.TestTools.BuildShadowsTask.dll"/>

Here I didn’t make any changes to the orginal .targets file. I import the BuildShadowTask as before. I will, in fact, use it, just in a round-about way (you’ll see what I mean in a bit). 

  <PropertyGroup>

    <CoreResolveTestReferencesTarget Condition=" '$(CoreResolveTestReferencesTarget)' == '' ">ExternResolveTestReferences</CoreResolveTestReferencesTarget>

  </PropertyGroup>

Here is where some of the magic happens. Remember that I said my solution is to shell out to MSBuild 3.5. Well, the .targets file that I’m creating will be included in both the MSBuild 4.0 and MSBuild 3.5 cases. That is, the initial build and the shelled out build will both use this same logic. I differentiate the two scenarios in this property. The condition on the property states “if the property isn’t set”. That is, I only set the property if it wasn’t previously set. In this case, it won’t be set when the .targets file is loaded from MSBuild 4.0. When I shell out to MSBuild 3.5, I’ll set it explicitly. From MSBuild 4.0 I’ll set it to ExternResolveTestReferences, but when it is called from MSBuild 3.5 it will be set to CoreResolveTestReferences.

  <PropertyGroup>

    <ResolveReferencesDependsOn>

      $(ResolveReferencesDependsOn);

      $(CoreResolveTestReferencesTarget)

    </ResolveReferencesDependsOn>

  </PropertyGroup>

Like the original .targets file, I needed to fit my targets into the dependency chain for the common build targets. We add the target in the value of the property CoreResolveTestReferencesTarget, which I defined above.

  <PropertyGroup>
    <ResolveTestReferencesDependsOn>
      ResolveReferences;
      $(CoreResolveTestReferencesTarget)
    </ResolveTestReferencesDependsOn>
  </PropertyGroup>

I also added a standard dependency property for the ResolveTestReferences target. This is because when you use the original .targets file, you can’t build just the ResolveTestReferences target, because it doesn’t define its dependencies. I needed to call the target directly, so I added the required dependencies.

  <Target Name="ResolveTestReferences" Condition="'@(Shadow)'!=''" DependsOnTargets="$(ResolveTestReferencesDependsOn)" />

The original ResolveTestReferences contained all the logic to call out to the BuildShadowTask. Since I need to use different logic between MSBuild 4.0 and MSBuild 3.5 I added the target dependencies and removed the logic. The condition just states that “if there are Shadow items”. In other words, only run the target if items are defined as Shadow, which are the test reference (.accessor) files.

  <Target Name="CoreResolveTestReferences" Condition="'@(Shadow)'!=''">

The CoreResolveTestReferences target is the one called from MSBuild 3.5. It has the same condition as the ResolveTestReferences target to avoid unnecessary invocations. This target will look a lot like the original ResolveTestReferences target. In fact the next bit is a direct copy:

    <BuildShadowTask

        CurrentResolvedReferences="@(ReferencePath)"

        CurrentCopyLocalFiles="@(ReferenceCopyLocalPaths)"

        Shadows="@(Shadow)"

        ProjectPath="$(ProjectDir)"

        IntermediatePath="$(IntermediateOutputPath)"

        SignAssembly="$(SignAssembly)"

        KeyFile="$(AssemblyOriginatorKeyFile)"

        DelaySign="$(DelaySign)">

      <Output TaskParameter="FixedUpReferences" ItemName="ReferencePath"/>

      <Output TaskParameter="NewCopyLocalAssemblies" ItemName="ReferenceCopyLocalPaths"/>

    </BuildShadowTask>

Just like the original ResolveTestReferences, a call out to the BuildShadowTask is done. The difference is in the next couple of lines:

    <WriteLinesToFile

        File="$(IntermediateOutputPath)ReferencePath.txt"

        Lines="@(ReferencePath)"

        Overwrite="true" />

    <WriteLinesToFile

        File="$(IntermediateOutputPath)ReferenceCopyLocalPaths.txt"

        Lines="@(ReferenceCopyLocalPaths)"
        Overwrite="true" />

All I’m doing here is writing the values of the ReferencePath and RefenceCopyLocalPaths items out to a text file. Note that there is a potential for data loss here, if the items have metadata attached to them. In my tests, I didn’t see any issues. The reason I write the values of these items out to a file is that I want to read them back into MSBuild 4.0 space. It was the easiest way I could think of to try to replicate the original behavior.

  </Target>

And, of course, this just closes the CoreResolveTestReferences target. Now onto the MSBuild 4.0 logic.

  <Target Name="ExternResolveTestReferences" Condition="'@(Shadow)'!=''">

The ExternResolveTestRefences target contains the logic for MSBuild 4.0 to shell out to MSBuild 3.5. It uses the same condition as seen on the ResolveTestRefences target.

    <Exec Command="&quot;$(MSBuildBinPath)\MSBuild.exe&quot; /t:ResolveTestReferences &quot;$(MSBuildProjectFullPath)&quot; /p:Configuration=&quot;$(Configuration)&quot; /p:Platform=&quot;$(Platform)&quot; /p:OutDir=$(OutDir) /p:CoreResolveTestReferencesTarget=CoreResolveTestReferences" />

Here is where we shell out to MSBuild 3.5. In MSBuild 4.0, I found that the MSBuildBinPath property is set to the target framework’s bin path, not the 4.0 bin path, so I was able to use that to invoke the correct version of MSBuild.exe.

If you are not familiar with the MSBuild command line arguments, the /t indicates the target or targets to run and /p defines a property. Since I set up the dependency chain for the ResolveTestRefences target, we can just tell MSBuild 3.5 to build that target. After the /t argument you’ll see that I pass the full path to the MSBuild project that is currently being built, which should be the unit test project. The rest of the arguments are properties that need set to maintain behavior. This is where you might find the need to do some customization. I’ll go over some of the properties and why I included them.

The Configuration property is the configuration that you want to build. A configuration is typically ‘Release’ or ‘Debug’ for .NET projects, but it can be customized. Anyway, here I just wanted to pass the currently building configuration on to MSBuild 3.5 so it builds to the right folders, etc.

The Platform property is the platform that you are targeting. For example, x86 or x64. For .NET projects, it will typically be ‘Any CPU’. Just like the Configuration property, I just wanted to maintain the same for MSBuild 3.5 since it often is used to construct output file paths, etc.

The OutDir property is set because it is a commonly overridden property for TFS Build builds. This problem was initially found from a customer that was trying to use TFS Beta 2 to build legacy projects. Well, TFS Build overrides the OutDir property in order to get all your projects to put their outputs into a single location. To get project references and such to resolve correctly, it is needed, although no outputs are placed there from my invocation of MSBuild 3.5.

Finally, the CoreResolveTestReferencesTarget is set to CoreResolveTestReferences. This is the key to making sure that the invoked MSBuild 3.5 uses the CoreResolveTestReferences target, which does the actual BuildShadowTask call. Don’t change this.

Ok, now the last thing to do is read back in those files that we wrote from the CoreResolveTestReferences task in MSBuild 3.5.

    <ReadLinesFromFile File="$(IntermediateOutputPath)ReferencePath.txt">

      <Output TaskParameter="Lines" ItemName="ReferencePath" />

    </ReadLinesFromFile>

    <ReadLinesFromFile File="$(IntermediateOutputPath)ReferenceCopyLocalPaths.txt">

      <Output TaskParameter="Lines" ItemName="ReferenceCopyLocalPaths" />
    </ReadLinesFromFile>

Simply, I read the lines right back into the properties that they were written from in MSBuild 3.5, but in MSBuild 4.0 space. With any luck, any other targets will have the expected values already in these item properties. This was the final key to maintaining the same behavior as the original ResolveTestReferences target.

    <Delete Files="$(IntermediateOutputPath)ReferencePath.txt" />
    <Delete Files="$(IntermediateOutputPath)ReferenceCopyLocalPaths.txt" />

For good measure, I go ahead and delete the intermediate files, since they mean nothing for incremental building.

  </Target>

And, of course, this ends the ExternResolveTestReferences target.

All that code is attached as Microsoft.TeamTest.4.targets. You’ll want to drop that file into your %ProgramFiles%\MSBuild\Microsoft\VisualStudio\v9.0\TeamTest folder. On an x64 system, put it in %ProgramFiles(x86)%\MSBuild\Microsoft\VisualStudio\v9.0\TeamTest.

Now, to get your unit test projects to pick up this .targets file instead of the original, you need to add one property to them. If you have a large project structure, then perhaps you have a single place that you can add it. If you have a large project structure and aren’t using .targets files for your common customizations (properties, custom build processes, etc) then I suggest you start looking into it.

The key is to get this property defined before you import Microsoft.CSharp.targets. If you are at all familiar with MSBuild project files, you won’t have too much trouble. For the rest of you, I’ll show you how to add it to a plain ol’ unit test project. Open the unit test project file (.csproj or .vbproj) in notepad or VS using the XML editor. Find the first set of properties, something like:

  <PropertyGroup>

    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>

    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>

    <ProductVersion>9.0.30729</ProductVersion>

    <SchemaVersion>2.0</SchemaVersion>

    <ProjectGuid>{9C571CAA-F9D2-4103-8B1E-5AB41E34C187}</ProjectGuid>

    <OutputType>Library</OutputType>

    <AppDesignerFolder>Properties</AppDesignerFolder>

    <RootNamespace>UnitTests</RootNamespace>

    <AssemblyName>UnitTests</AssemblyName>

    <TargetFrameworkVersion>v3.5</TargetFrameworkVersion>

    <FileAlignment>512</FileAlignment>

    <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

  </PropertyGroup>

Add this property somewhere in that property group:

<MsTestToolsTargets>$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v9.0\TeamTest\Microsoft.TeamTest.4.targets</MsTestToolsTargets>

Save the file, and you’re done. Go build from MSBuild 4.0. You should see that the generated accessor assemblies are linked to the 2.0 CLR and not the 4.0 CLR. You can verify by opening the assembly in ILDasm.exe. Open the MANIFEST node.

What you should see:

.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 2:0:0:0
}

What you should not see:

.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 4:0:0:0
}
.assembly extern mscorlib as mscorlib_2
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 2:0:0:0
}

Conclusion

You can in fact build 2.0 linked accessor assemblies using MSBuild 4.0 Beta 2. However, this is only a workaround. You may need to customize it to your environment or needs.

I hope this helps some people out. Please do leave feedback if you find errors or if you just want to give a big thumbs up (or down).

Downloads

http://cid-6f4c66b0ee56cd90.skydrive.live.com/self.aspx/Public/Building%20VS%202008%20Unit%20Test%20Projects%20in%20MSBuild%204.0%20^5Beta%202^6/Microsoft.TeamTest.4.zip

Jim Lamb wrote a good post about Enabling Symbol and Source Server Support in TFS Build 2010 Beta 1. His post gives a great overview of the requirements to get it working out of the box (it’s very easy!), while this post aims to help the beginner understand how it works, to elaborate on some details of the features, and to identify what customizations are available.

Note that the source server and symbol server features complement each other nicely, but neither requires the other.

Source Server Features

If you are unfamiliar with Microsoft’s Source Server features, it really is just metadata (source index) added to debugging symbols. A debugger with source server support can read the metadata and locate the correct versions of files and download them as needed during a debugging session. The metadata is contained in a “stream” of a PDB file and keeps the server location, the server path, and the version for each source file referenced in the symbols. That’s it. This process of adding metadata to debugging symbols is sometimes referred to as “source indexing”.

The TFS Build 2010 implementation largely follows the lead of ssindex.cmd from the Debugging Tools for Windows package. In fact, some of the tools shipped with Debugging Tools for Windows are redistributed with TFS 2010 so that the Debugging Tools don’t need to be separately installed. This makes enabling source server support extremely easy. All you need to do is set the Index Sources build parameter to true for the default build process template. You can do this for any build definition that uses the default template by changing the default value for the parameter, or you can change it in the Queue Build dialog when you queue a build (for a definition using the appropriate template).

The source indexing is done by the IndexSources activity provided by TFS Build. It takes only one argument called FileList, which is the list of symbol files (PDBs) to add a source index to. By default, all PDBs under the output directory are indexed.

IndexSources uses the current build (the one that is building) and workspace mappings to determine the version control information it needs for the sources, and doesn’t let you change how it uses that information. The limitation here is that if you are building sources that come from multiple workspaces (not recommended) then your source index may be incomplete.

Customizing the Source Index

Consider this a warning that messing with the source index outside of using the IndexSources activity is not supported by TFS.

One of the tools that comes in Debugging Tools for Windows is pdbstr.exe. Its sole purpose is to read and write “streams” from PDB files. One situation you may find yourself in is that your TFS installation is accessible from the web, or from anywhere that requires a different URL than the one that the build machine uses, you may need to read out the source index, modify the TFS URL, and then write the source index back.

You can read the source index to a file using this command:

pdbstr.exe -r -p:"<pdb path>" -i:"<index file>" -s:srcsrv

Edit the VSTFSSERVER variable value to point to the appropriate external URL. Then write the index back using:

pdbstr.exe -w -p:"<pdb path>" -i:"<index file>" -s:srcsrv

If any readers have other specific requirements for source indexing, please do let me know and I’ll try to address them.

Symbol Server Features

A symbol server is a Windows network share or folder that has a particular format, managed by symstore.exe, another tool shipped with Debugging Tools for Windows. If you direct a supported debugger to that share to find symbols, it should be able to automatically find the right version for the binaries that you are debugging (providing they exist in the share somewhere).

TFS Build 2010 provides support to add symbols to the symbol server share during a build. This is typically done after source indexing (if source indexing is being used) so that someone debugging a build for which they don’t (directly) have symbols or sources can seamlessly step through code and examine call stacks and variables.

To add symbols to a symbol server share, TFS Build provides the PublishSymbols activity. The activity works out-of-the-box by setting the Path to Publish Symbols build process parameter. You should set it to a UNC path, even if you are publishing to a local machine folder, as this feature was aimed at the “typical” user.

Customizing Symbol Publishing

Your customization options for symbol publishing is pretty wide open. Since PublishSymbols is really just a wrapper around the InvokeProcess activity (also shipped with TFS Build) which calls symstore.exe with the “add” command, you can provide additional command line arguments using the CommandLineArguments argument. Remember, these are additional command line arguments. The StorePath and ProductName activity arguments are required, so you shouldn’t include them in CommandLineArguments. Also, the files to publish are provided to symstore.exe as a temp file that contains a list of files, which is generated by the files provided in the FileList argument. Examples of what you may add to CommandLineArguments are the /3 and /p options. Those are simple examples, refer to symstore.exe documentation on the web for more.

Of course, you could always modify the template and replace PublishSymbols with InvokeProcess in circumstances where PublishSymbols doesn’t give you enough flexibility, but note that PublishSymbols tags the build with information about the symstore.exe transaction so that symbols can be deleted when a build is deleted. You could try to replicate that information yourself, but I won’t detail that here.

The SharedResourceScope Activity

One more thing to note about symbol server publishing is the use of the SharedResourceScope activity in the build process template. The purpose is to make sure that concurrent instances of symstore.exe aren’t adding symbols at the same time, as the tool doesn’t support concurrent access to a symbol server share. SharedResourceScope uses the Team Foundation Server to control access to an arbitrarily named resource, in this case the share. That way, if multiple builds are trying to publish symbols at the same time, the requests are queued and only one will publish at a time, while the others wait (instead of fail due to file access errors or “step on each others’ toes”). PublishSymbols does not care about shared resource locks, but it is contained within the SharedResourceScope, so won’t be executed until the lock is appropriately acquired.

Conclusion

With Team Foundation Server 2010 you can boost the efficiency across your dev and test teams using the provided source server and symbol server features out-of-the-box. Yet, for the experienced teams with more precise requirements, you have a lot of options for customization.

Read More ...

In Team Foundation Server 2008 you could delete builds both from the Build Explorer, using the context menu on one or more builds, and from the command line, using TfsBuild.exe. Also, retention policy would automatically delete builds as specified. Once you deleted the build, you no longer were able to see the build in the Build Explorer, as it no longer "exists", and TfsBuild.exe's delete command will complain that the build doesn't exist. In Team Foundation Server 2010, you may be surprised to find that, although you don't see the build in Build Explorer, your data still exists in the database. Well, some of it.

There is more to the story. Another new feature in TFS 2010 is that you can specify what gets deleted when you delete a build, including the drop location (for binaries and other outputs), the version control label, test results, symbols (on a symbol server), and the build details (the actual database records related to the build). Until you delete the build details, your build will remain visible in Build Explorer. This article addresses actions taken after the build detail is deleted.

There are several reasons for TFS 2010 to keep build data around after the user deletes it, which I'll address in the following sections.

Build Information that Doesn't Get Deleted

So, when you delete the build details of a build, what doesn't get deleted from the database?

• The build's detail record. This gets a flag set on it that denotes it as a "deleted" build, but is otherwise intact.
• The build's top level information nodes and some specific node types. There are several pieces of information that are stored in information nodes that are critical for post-deletion activities. A few are:
  • Associated changesets, used for warehouse rebuilding.
  • Configuration summaries and build projects, used for warehouse rebuilding.
  • Symbol store transaction information, used to delete a build’s symbols.
• The build's outputs. These are used to query build information by binary signatures of an assembly or native image.
• The build's workspace. Can be queried to reproduce a build or for informational purposes.

You may wonder how much space this uses per build. Well, it will vary greatly depending on the size and complexity of your project, so I won't try to answer that here. I'll try to do some analysis and give some rough estimates in a later post.

Rebuilding the Warehouse

The warehouse is where reporting data is archived in order to keep historical data as well as to keep the production databases smaller/cleaner for performance and efficiency reasons. I won't go into any details about how or why you may rebuild the warehouse, since I don't have the expertise in that area.

Deleting Remaining Build Artifacts

Let's say you've been deleting builds via retention policy, but you haven't been deleting the build symbols. This may be a useful scenario if you have builds installed or lingering around for longer periods of time than you want to keep builds for. But eventually, your symbol store will become very bloated, so you'll want to clean it up. One method to clean it up is to view the logs in the store and decide which symbols to delete that way. Another way is to let TFS do it for you. You can specify on the command line:

tfsbuild.exe delete /server:http://tfsat:8080/collection0 /builddefinition:"Adventure Works\Nightly Build" /daterange:~2008-12-31 /deleteoptions:Symbols

This particular command will connect to TFS at http://tfsat:8080/collection0 (/server:http://tfsat:8080/collection0) and delete symbols (/deleteoptions:Symbols) for all builds older than December 31, 2008 (/daterange:~2008-12-31) in the build definition called "Nightly Build" on the team project "Adventure Works" (/builddefinition:"Adventure Works\Nightly Build"). There are several variations to the delete command, which you can explore in documentation or via command line help. The /deleteoptions parameter takes one or more of:

• Details
• Label
• DropLocation
• TestResults
• Symbols
• All

To combine multiple, you can separate them by a comma, like:

/deleteoptions:Details,DropLocation

Remember, you can delete any artifacts (that haven't been previously deleted) even after you delete the build details. That is because a limited amount of data is still stored in the database, including everything needed to delete all the artifacts.

Matching Binaries to a Build

The concept here is that you can query TFS to ask it which build created a particular binary. This is a not-well-documented feature, but it can be pretty useful. The way this works is that during a build, there is an activity that associates build outputs to the build, called AssociateBuildOutputs. What it does is read the PE (Portable Executable) image headers of a managed assembly or native binary and extracts a couple of identifying pieces of information. It stores those in an indexed table of the database. Then, when you have a binary, but for whatever reason don't know specific information about it's version/build, you can read the same information from the PE headers and query the database. This can be done for any build that exists or that has already been deleted (even if the build details were deleted).

Unfortunately, there is no tool that supports this from the command line or Visual Studio. You'll have to write some code, but it isn't very hard. The TFS Build client libraries will do the header reading for you, so don't worry. Here is an example:

using Microsoft.TeamFoundation.Build.Client; 

static IBuildDetail GetBuildForBuildOutput(IBuildServer buildServer, string outputPath)
{
    // Return values of the GetSignature function
    Guid guid;
    int age;
    
    // GetSignature returns false if it failed
    // to read the executable file
    if (!ModuleSignatureReader.GetSignature(outputPath, out guid, out age))
    {
        return null;
    }
    
    // GetBuildForBuildOutput returns null if
    // an appropriate build isn't found
    return buildServer.GetBuildForBuildOutput(guid, age);
}

All we are doing here is wrapping a couple functions so we can query by the assembly's path. ModuleSignatureReader.GetSignature will open the file, so it needs IO read access to it. It returns true for success and false for failure. IBuildServer.GetBuildForBuildOutput then calls out to TFS to do the query and return the result. It returns null if a build can't be found for the output.

Getting a Build's Workspace Mappings

TFS Build needs a particular workspace setup to build, just like any dev working in the source code. Often this will be the same as any dev's workspace, but sometimes it will include more or less in order to satisfy specific build requirements (such as building a complete set of products which depend on each other on a nightly basis, or building just a piece of the source tree for every checkin). You can query for the particular workspace mappings used from any build.

You can do this for any build normally, but since this article focuses on deleted builds, we'll be looking for a deleted build and then getting the workspace mappings. There are only a couple methods that take a QueryDeletedOption. The IBuildServer.QueryBuildsByUri function takes it, and any functions that take IBuildDetailSpec do, by way of the IBuildDetailSpec.QueryDeletedOption property. Here is what that QueryDeletedOption enum looks like:

public enum QueryDeletedOption
{
    ExcludeDeleted = 0,
    IncludeDeleted = 1,
    OnlyDeleted = 2,
}

Keep in mind that when we talk about "deleted" builds, we really mean the build details (database records). That still applies with this enumeration. ExcludeDeleted means don't include any deleted builds in the results. IncludeDeleted means go ahead and query deleted builds as well as not-deleted builds. OnlyDeleted means just what it says, only query for deleted builds.

Again, the command line and Visual Studio client don't have access to this data. You'll have to write code:

// Query the desired build, using QueryDeletedOption.Deleted
IBuildDetail build = buildServer.QueryBuildsByUri(new Uri[] { buildUri }, null, QueryOptions.None, QueryDeletedOption.Deleted);

// Get the workspaces used by the build
IWorkspaceInstance[] workspaces = build.GetWorkspaceInstances();

// Iterate through the workspaces
foreach (IWorkspaceInstance workspace in workspaces)
{
    Console.WriteLine("Workspace:");
    foreach (IWorkspaceInstanceMapping mapping in workspace.Mappings)
    {
        Console.WriteLine("[{0}] {1}", mapping.MappingType, mapping.ServerItem);
    }
}

Note that there are circumstances where multiple workspaces may have been used by a build (likely a custom build) so you may have multiple workspace instances.

Destroying Builds

Note: Destroying builds is not for the faint of heart. Once a build is destroyed it cannot be recovered.

Destroying builds is about cleaning up your database. I believe in 9 out of 10 cases there is no need to destroy builds … ever. However, if your projects do a lot of builds, destroying old builds can save space and even increase performance.

When you destroy builds, you are not deleting build artifacts, such as version control labels and test results. Destroying a build permanently deletes the information from the database. Typically, if you are destroying builds, you will want to delete all the build artifacts prior to doing so. If you intend to keep the build artifacts, that is fine, too.

From the command line, you can destroy builds individually, by date range, or a couple other ways. See the documentation or command line help for more information. I envision the destroy command being used to destroy old builds. You may try something like:

tfsbuild.exe delete /server:http://tfsat:8080/collection0 /builddefinition:"Adventure Works\Nightly Build" /daterange:~2008-12-31 /deleteoptions:All
tfsbuild.exe destroy /server:http://tfsat:8080/collection0 /builddefinition:"Adventure Works\Nightly Build" /daterange:~2008-12-31

Of course, the delete command is optional. I use it to delete all the build artifacts prior to destroying the builds. These command lines are very similar to the one earlier in this article. Basically, we are deleting all build artifacts and then destroying the database records for builds older than December 31, 2008 for the "Nightly Build" build definition in the "Adventure Works" team project.

You can also destroy builds from the TFS Build client libraries. You'd do it something like this:

// Create a build detail spec to query all deleted
// builds prior to December 31, 2008
IBuildDetailSpec spec = buildServer.CreateBuildDetailSpec(definition);
spec.MaxChangedTime = DateTime.Parse("2008-12-31");
spec.QueryDeletedOption = QueryDeletedOption.IncludeDeleted; 

// Query the builds
IBuildDetail[] builds = buildServer.QueryBuilds(spec).Builds;

// Delete any remaining artifacts for all the builds
// (Note: this can take a long time, so do it one at a time)
foreach (IBuildDetail build in builds)
{
    build.Delete(DeleteOptions.All);
}

// Now destroy the builds
buildServer.DestroyBuilds(builds);

Again, the buildServer.DeleteBuilds call is optional, and is just used to delete all the build artifacts prior to destroying the build data.

Conclusion

As you can see there are several things to do with deleted builds. A lot of them require getting down and dirty with the command line or custom coded tools, but there is nothing that is terribly difficult to manage. The new /daterange option for the delete and destroy commands of TfsBuild.exe will be a nice addition to those of you that like to use the baked tools, but I recommend PowerShell scripts to give you better control of your scheduled maintenance tasks.

There are some other potential bug fixes on the way, too.

When it comes interop, .NET has solved most of the hard problems. However, if you've worked with COM interfaces that return or take PROPVARIANTs as parameters, you may have noticed that there isn't much support for this union'd struct type. I came across one such interface, and needed a good way to interop with its PROPVARIANT parameters. In fact, the implementation I was working with returned BLOBs, which break all sorts of basic implementations of variant interop.

Of course, PROPVARIANTs are used in other places than just COM. You may find yourself using P/Invoke with PROPVARIANT params, as well.

Background

The PROPVARIANT reuses a lot of definitions of the VARIANT/VARIANTARG types. In fact, they are nearly identical. I guess the difference is around their intended use. That is, VARIANTARG would be used to pass a set of self described arguments to a function, where the caller maintains the memory management role before and after the call. A PROPVARIANT is ideal for out params, where the caller gets an instance of the value that it needs to maintain the memory of, but the callee still maintains its own value's memory.

.NET actually has some support for interop of VARIANT types. For example, if you define this method:

.NET will see the MarshalAs attribute set to use UnmanagedType.Struct in combination with the object type for the parameter and assume the variable you are marshaling is a VARIANT. This works for types like int, short, byte, float, double, but not for types like BLOB. In this case you'll get a runtime error, because .NET considers the VARIANT invalid. If you'd like to know more about how .NET decides what an invalid VARIANT is, take a look at Adam Nathan's post about debug probes.

The Union

The PROPVARIANT has an interesting structure. For the most part, it is a simple struct, with a field that identifies the type of data, plus a union field that contains the data itself. For those of you .NET developers that don't know what a union is, I'll give you a brief intro.

In a C/C++ type you can define a field as a union. Here is a simple example:

// C++
typedef struct tagMYSTRUCT
{
    union
    {
        int iVal;
        BYTE bVal;
    };
} MYSTRUCT;

In this struct I have defined two fields. Unlike a normal struct, though, I have pulled the two fields into a union. That union says that the two values share the same memory. So, the size of the union member is the max of the sizes of the fields defined within it. In this case, sizeof(int) > sizeof(BYTE), so the union members size equals sizeof(int).

The Native PROPVARIANT Structure

Let's take a look at the structure of the native PROPVARIANT:

This structure has an extra level of confusion... it nests a union in a struct in a union. This is because of that DECIMAL value you'll see at the end. The DECIMAL field actually overlaps the VARTYPE field (via the outer union). This is because the PROPVARIANT's size is 16 bytes (on a 32-bit architecture) before the DECIMAL member, which is 16 bytes in itself. If the DECIMAL member were within the inner union, the size of the PROPVARIANT would grow to 20 bytes, which is no longer compatible with VARIANTs. I ignore the DECIMAL field in my implementation, because I've not run across a need for it.

Most of the types in the PROPVARIANT are simple, including 1/2/4/8 byte integers, 4/8 byte floating point numbers, booleans, and pointers. There are a couple of structs to take note of. BSTRBLOB and BLOB are structs that have two members, a 4-byte integer, and a pointer. These will be considered when we come up with our .NET structure layout.

Creating a Managed PROPVARIANT

There have been some attempts at laying out a managed PROPVARIANT by using LayoutKind.Explicit and the FieldOffsetAttribute. These work for marshaling an unmanaged struct to a managed struct, but I've seen issues going the other way. Also, because the fields of a struct in .NET will never share memory, you may end up with hundreds of bytes in your managed struct, which is passed by value! Other implementations use an IntPtr for the union member, but this doesn't allow for accessing all 8 bytes of the union member, at least not on a 32-bit architecture.

My implementation takes a different approach. I use an IntPtr for the first 4 bytes of the union, but I use an extra int field to access the second 4 bytes. That gives me access to all 8 bytes of the union. Plus, on a 64-bit architecture, that provides me with 12 bytes, because the size of the IntPtr becomes 8 bytes. Note that the size of the largest member of the union on a 64-bit architecture is 12 bytes, that is why I used one IntPtr and one int, instead of two IntPtrs which would result in 16 bytes for the union member.

The definition of my managed struct looks like this:

NOTE: The vt member value is from the System.Runtime.InteropServices.ComTypes.VarEnum enumeration.

NOTE: I tried to code it to work on 64-bit architectures, but I don't have one to test on, so I'll leave it to my readers to tell me what does/doesn't work with regards to that. This struct will have a size of 16 bytes on a 32-bit architecture, and a size of 20 bytes on a 64-bit architecture.

This completes the implementation for marshaling purposes. You could use this struct to pass a PROPVARIANT to a method, or get a PROPVARIANT as an out parameter, etc. For example, an unmanaged method may like this:

The managed import might look like this:

or

// C#
void GetProp(ref PropVariant pval);

The standard .NET marshaler will handle these interop calls just fine without any special marshaling attributes.

However, this doesn't make for easy examination of the data value of the variant.

Accessing the Data of the Managed PropVariant

What I've done is taken the union members of the native struct and created them as private member properties of my managed struct. Each property does the appropriate bit twiddling to get its value from the IntPtr field of the struct, and the trailing int field when necessary. Here are a couple of the properties:

The BitConverter is part of the framework, and comes in handy when you are working with bits that you don't want to convert by value (ie. (float)1 results in a float of value 1, not a float with bits 0x00..0001).

I wrote a GetDataBytes function to help get the needed bytes from the union part of the structure:

Now, based on the vt member, we can determine which property to use to get the real value. This can be simplified into a single public property:

All I do here is determine the type of the data, then return the proper value, referencing my private properties for conversions. For some of the types, I do some special handling. For example, the string types need to allocate managed strings from the native strings, and for the VT_UNKNOWN type I get the COM object that the IUnknown pointer references. Also, the VT_BLOB type is handled so that it gets the size of the data, then allocates a managed byte array to copy the data into. For this type, we have to take into account the architecture.

Cleaning Things Up

Because I don't rely on marshaling to clean up unmanaged memory, I need to provide a mechanism to make sure it gets cleaned up. Unfortunately, I can't add a finalizer to the struct to ensure its proper deallocation, but I can provide a method that depends on the user to call when finished with the value:

    [DllImport("ole32.dll")]
    private extern static int PropVariantClear(ref PropVariant pvar);

    public void Clear()
    {
        // Can't pass "this" by ref, so make a copy to call PropVariantClear with
        PropVariant var = this;
        PropVariantClear(ref var);

        // Since we couldn't pass "this" by ref, we need to clear the member fields manually
        // NOTE: PropVariantClear already freed heap data for us, so we are just setting
        //       our references to null.
        vt = (ushort)VarEnum.VT_EMPTY;
        wReserved1 = wReserved2 = wReserved3 = 0;
        p = IntPtr.Zero;
        p2 = 0;
    }

NOTE: I could have implemented the structure as a class in order to implement a finalizer and IDisposable, then require the MarshalAs attribute to marshal as an UnmanagedType.LPStruct, but that would require the user to initialize the class before use. The usage pattern would also be changed for reusing the object for multiple calls. If you were to reuse the object, the fields would be overwritten without being properly deallocated first. This can happen when using a struct, too, but by using the pattern I chose, you are always expected to call the Clear function, whereas with a class, you only have to call it when reusing the object. The inconsistency is what made me settle on using a struct.

Using the PropVariant struct

Using the struct is easy. The usage is similar to the Dispose pattern, however we can't utilize the using keyword. Using the above example method, we can call it like this:

If you don't call Clear, and the PROPVARIANT contains a pointer to some data, you *will* have a memory leak.

For reusing the variable, you can wrap all your calls in one try..finally block. This is because calling the Clear method on an empty PropVariant does not throw an exception:

Source Code

I've provided source for the PropVariant struct as a single C# code file. I've also provided a project that has sample usage, using a C++ COM project, plus a test project.

Of course, PROPVARIANTs are used in other places than just COM. You may find yourself using P/Invoke with PROPVARIANT params, as well.

Read More ...

This is part of a 3 part series:

1. Shell Style Drag and Drop in .NET (WPF and WinForms)
2. Shell Style Drag and Drop in .NET - Part 2
3. Shell Style Drag and Drop in .NET - Part 3

Introduction

In Part 1, Shell Style Drag and Drop in .NET (WPF and WinForms), I opened up the discussion about implementing a nice Shell style drag image, like that of Windows Explorer, in C#. This involved exposing a couple of COM interfaces to .NET, as well as implementing the COM IDataObject interface. In Shell Style Drag and Drop in .NET - Part 2 I took it to the next step, implementing some extension methods and a helper class. These made it easy to use the COM interfaces to achieve Shell integration.

In Part 3, I will open the door even wider. My aim is to achieve complete Shell integration. First, I will look at adding a drop description. This is the little preview text that tells you what you would do by dropping at the current location. It is a dynamic text with an icon, indicating whether you are moving, copying, linking, etc. Then I'll look at handling all the drag source plumbing in a consistent fashion through the use of a new class, the DragSourceHelper. Unlike the DropTargetHelper class from Part 2, the DragSourceHelper actually has a decent amount of work to do.

Background

In Parts 1 and 2 I utilized the IDragSourceHelper COM interface exposed by the Windows Shell. In this part we will take a look at IDragSourceHelper2, which inherits IDragSourceHelper and exposed an additional function, SetFlags, for allowing drop descriptions.

• IDragSourceHelper2 - Exposes a method that adds functionality to IDragSourceHelper. This method sets the characteristics of a drag-and-drop operation over an IDragSourceHelper object.

The interface, like IDragSourceHelper, is implemented by the system DragDropHelper class, which is CoCreated using CLSID_DragDropHelper.

The Solution

The solution for this post is quite large, because we are bringing everything together for a complete solution. The solution comes in several parts, in each of which we'll address a major requirement for the overall solution.

Finishing the Implementation with Drop Descriptions

To get drop descriptions working, I need to set a flag on the drag image manager. To do this, I use the IDragSourceHelper2 interface:

    [ComVisible(true)]
    [ComImport]
    [Guid("83E07D0D-0C5F-4163-BF1A-60B274051E40")]
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    public interface IDragSourceHelper2
    {
        void InitializeFromBitmap(
            [In, MarshalAs(UnmanagedType.Struct)] ref ShDragImage dragImage,
            [In, MarshalAs(UnmanagedType.Interface)] IDataObject dataObject);

        void InitializeFromWindow(
            [In] IntPtr hwnd,
            [In] ref Win32Point pt,
            [In, MarshalAs(UnmanagedType.Interface)] IDataObject dataObject);

        void SetFlags(
            [In] int dwFlags);
    }

The IDragSourceHelper2 interface provides the SetFlags function. According to MSDN, there is only one flag accepted by this function, and that is DSH_ALLOWDROPDESCRIPTIONTEXT which is defined in shlobjidl.h as 0x1. We will utilize this method solely to enable drop descriptions.

I also need to introduce a native structure, DROPDESCRIPTION, as a managed structure:

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Size = 1044)]
    public struct DropDescription
    {
        public int type;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
        public string szMessage;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
        public string szInsert;
    }

Note that the size of the szMessage and szInsert fields is limited to 260 Unicode characters during marshaling. When I set the drop description, this is the structure that I need to set onto the IDataObject for the drag image manager to read and render from.

I also need a new enum, which I've introduced separately in the System.Windows and System.Windows.Forms namespaces for ease of use:

    public enum DropImageType
    {
        Invalid = -1,
        None = 0,
        Copy = (int)DragDropEffects.Copy,
        Move = (int)DragDropEffects.Move,
        Link = (int)DragDropEffects.Link,
        Label = 6,
        Warning = 7
    }

Not much to say here. These are the values for the "type" field on the DropDescription structure. Many of the values correlate directly to the DragDropEffects enum.

To set the drop description on the data object, I introduce an extension method to the SWF and WPF IDataObject interfaces. As in previous parts, I will use the SWF implementation in my examples:

        public static void SetDropDescription(this IDataObject dataObject, DropImageType type, string format, string insert)
        {
            if (format != null && format.Length > 259)
                throw new ArgumentException("Format string exceeds the maximum allowed length of 259.", "format");
            if (insert != null && insert.Length > 259)
                throw new ArgumentException("Insert string exceeds the maximum allowed length of 259.", "insert");

            // Fill the structure
            DropDescription dd;
            dd.type = (int)type;
            dd.szMessage = format;
            dd.szInsert = insert;

            ComTypes.ComDataObjectExtensions.SetDropDescription((ComTypes.IDataObject)dataObject, dd);
        }

Here I accept a DropImageType enum value, a format string and an insert string. See the documentation for DROPDESCRIPTION on MSDN for details about format and insert, but basically know that you can specify a format like "Move to %1" where "%1" gets replaced with the insert string upon rendering. This is to distinguish the verb from the location, for example. The drag image manager renders the parts of the drop description slightly different to achieve the distinction.

This method leaves some of the work to the ComDataObjectExtensions.SetDropDescription method, which is actually an extension method for the COM IDataObject interface. I separated the logic like this because my SWF and WPF implementations can both take advantage of it. Let's look at that method now:

        public static void SetDropDescription(this IDataObject dataObject, DropDescription dropDescription)
        {
            ComTypes.FORMATETC formatETC;
            FillFormatETC(DropDescriptionFormat, TYMED.TYMED_HGLOBAL, out formatETC);

            // We need to set the drop description as an HGLOBAL.
            // Allocate space ...
            IntPtr pDD = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(DropDescription)));
            try
            {
                // ... and marshal the data
                Marshal.StructureToPtr(dropDescription, pDD, false);

                // The medium wraps the HGLOBAL
                System.Runtime.InteropServices.ComTypes.STGMEDIUM medium;
                medium.pUnkForRelease = null;
                medium.tymed = ComTypes.TYMED.TYMED_HGLOBAL;
                medium.unionmember = pDD;

                // Set the data
                ComTypes.IDataObject dataObjectCOM = (ComTypes.IDataObject)dataObject;
                dataObjectCOM.SetData(ref formatETC, ref medium, true);
            }
            catch
            {
                // If we failed, we need to free the HGLOBAL memory
                Marshal.FreeHGlobal(pDD);
                throw;
            }
        }

In this method I marshal the structure to an HGLOBAL and then set the data on the COM IDataObject instance.

Ok, there is one last thing to do, which is a bit more obscure. For the drag image manager to properly re-render the drag image with changing drop descriptions, I need to invalidate the drag image by sending a Window message to the drag window. The drag manager provides the drag window's HWND in the data format "DragWindow" on my IDataObject, so I can grab that and call the SendMessage Win32 API:

        private const uint WM_INVALIDATEDRAGIMAGE = 0x403;

        public static void InvalidateDragImage(IDataObject dataObject)
        {
            if (dataObject.GetDataPresent("DragWindow"))
            {
                IntPtr hwnd = GetIntPtrFromData(dataObject.GetData("DragWindow"));
                PostMessage(hwnd, WM_INVALIDATEDRAGIMAGE, IntPtr.Zero, IntPtr.Zero);
            }
        }

        private static IntPtr GetIntPtrFromData(object data)
        {
            byte[] buf = null;

            if (data is MemoryStream)
            {
                buf = new byte[4];
                if (4 != ((MemoryStream)data).Read(buf, 0, 4))
                    throw new ArgumentException("Could not read an IntPtr from the MemoryStream");
            }
            if (data is byte[])
            {
                buf = (byte[])data;
                if (buf.Length < 4)
                    throw new ArgumentException("Could not read an IntPtr from the byte array");
            }

            if (buf == null)
                throw new ArgumentException("Could not read an IntPtr from the " + data.GetType().ToString());

            int p = (buf[3] << 24) | (buf[2] << 16) | (buf[1] << 8) | buf[0];
            return new IntPtr(p);
        }

You won't find WM_INVALIDATEDRAGIMAGE in documentation, its value is actually WM_USER + 3. I don't remember where I found this, but it works. Would be interesting to know if there are other messages that the drag window accepts to get around some of the problems I'll touch on later.

This InvalidateDragImage method should be called by your drag source's GiveFeedback event handler. I'll walk you through a more complete GiveFeedback event handler later when I talk about the DragSourceHelper class.

Drop Descriptions Summary

Well, that's that. With these new foundations, I can enable and set drop descriptions for my drag images in my application. To enable the drop descriptions, I need to make sure to call SetFlags with an argument of 1 before I set the drag image via IDragSourceHelper.InitializeFromBitmap or IDragSourceHelper.InitializeFromWindow. To set the drop description, during DragEnter or DragOver events I can call e.Data.SetDropDescription with my desired arguments. The last thing to remember is that the drag source needs to listen to the GiveFeedback event and invalidate the drag window.

What Happened to the Managed SetData?

In Parts 1 and 2 when I implemented the COM IDataObject and set it as the inner data store for the SWF and WPF DataObjects, I forgot to address one issue. Both the SWF and WPF versions of DataObject implement both the respective IDataObject and the COM IDataObject. The reason I had to implement the COM IDataObject myself is that the .NET DataObjects don't allow setting data by calling the COM IDataObject.SetData method. They throw a COMException with E_NOTIMPL, which is basically a NotImplementedException. That prevented me from being able to use the .NET DataObjects because the Shell DragDropHelper requires the ability to call that COM callable method to store information for the drag image manager. However, the .NET DataObjects do allow you to provide a COM IDataObject to the constructors, and then would allow me to pass COM calls to the wrapped object directly. That is exactly how I implemented my solution.

(Note: I use ".NET SetData" and "COM SetData" to distinguish the SetData methods exposed by the .NET IDataObject and COM IDataObject respectively. The .NET SetData method is the one you, as a .NET developer, usually call. The COM SetData method is the one that is exposed by the COM IDataObject and is COM-callable. Also, where I've used SWF and/or WPF in the past, I will summarize as just ".NET". The two implementations of data objects are very very similar. So, when I say ".NET IDataObject interface(s)" I mean SWF or WPF, they are pretty much interchangable.)

Well, that put a new block on me. Now, the .NET DataObject that wrapped my COM IDataObject doesn't allow a .NET SetData call. It throws an InvalidOperationException. It is actually an internal converter class that throws the exception. The internal converter class implements the .NET IDataObject interface, and receives the .NET SetData call, but does nothing except throw an exception. The converter class gets created when I pass the COM IDataObject to the constructor of the .NET DataObject, which holds onto it as an internal data store, instead of the default internal data store. I drew up a diagram to show where my .NET SetData calls get blocked:

 
COM IDataObject Wrapped by .NET DataObject

Notice that there is a converter data object wrapping the COM IDataObject, and that is where the .NET SetData call fails. All other GetData and SetData calls go through as expected.

So, why not just implement the .NET IDataObject as well? Well, I certainly went down that path. Unfortunately, when you call DoDragDrop, which accepts the .NET IDataObject interface, as opposed to the DataObject class, somewhere along the line it gets wrapped by a .NET DataObject anyway. This wouldn't be a big deal, except there is another problem. The .NET DataObject classes seem to be less robust than I think they were intended. When I pass an instance of a class that implements both the .NET IDataObject interface as well as the COM IDataObject interface to the constructor, the .NET DataObject recognizes it as a .NET IDataObject and references it directly as its internal data store. That's great! Right? Well, no. The COM SetData method of the .NET DataObjects, as you can see in the above diagram, get a direct reference to the COM IDataObject of the internal converter data object class. Well, if the internal data store isn't of that type, then it will throw an exception, the same one I saw that made me implement the COM IDataObject interface to begin with. Here is a diagram showing this situation:

.NET/COM IDataObject Wrapped by .NET DataObject

As you can see, by the .NET DataObject classes not noticing that the internal data store is a COM IDataObject as well as a .NET IDataObject, I can't utilize both versions of SetData. So now what? Well, we only really have one choice. Because we require the COM SetData to function, we need to accept the fact that the .NET SetData won't. At least, not directly. Luckily, I have complete control, as the creator of the DataObject, and so I'll just write a new extension function, SetDataEx, which will accept managed data, marshal it to an HGLOBAL, and set it using the COM SetData. Let's get to it!

I don't need a completely robust solution, because in many cases, I can use the .NET DataObject classes as a converter. But I'll come back to that in a bit. For now, I need a function that takes managed data, marshals it, and sets it to the COM IDataObject using the COM SetData method. This seems like a great candidate for an extension method on the COM IDataObject interface, so that's exactly how I'll implement it. Note, this is not the same as the SetDataEx function I mentioned, which will use this method internally:

        public static void SetManagedData(this IDataObject dataObject, string format, object data)
        {
            // Initialize the format structure
            ComTypes.FORMATETC formatETC;
            FillFormatETC(format, TYMED.TYMED_HGLOBAL, out formatETC);

            // Serialize/marshal our data into an unmanaged medium
            ComTypes.STGMEDIUM medium;
            GetMediumFromObject(data, out medium);
            try
            {
                // Set the data on our data object
                dataObject.SetData(ref formatETC, ref medium, true);
            }
            catch
            {
                // On exceptions, release the medium
                ReleaseStgMedium(ref medium);
                throw;
            }
        }

This method really handles just the conversion of a string format to the FORMATETC structure. The meat of the marshaling is in the GetMediumFromObject function:

        private static void GetMediumFromObject(object data, out STGMEDIUM medium)
        {
            // We'll serialize to a managed stream temporarily
            MemoryStream stream = new MemoryStream();

            // Write an indentifying stamp, so we can recognize this as custom
            // marshaled data.
            stream.Write(ManagedDataStamp.ToByteArray(), 0, Marshal.SizeOf(typeof(Guid)));

            // Now serialize the data. Note, if the data is not directly serializable,
            // we'll try type conversion. Also, we serialize the type. That way,
            // during deserialization, we know which type to convert back to, if
            // appropriate.
            BinaryFormatter formatter = new BinaryFormatter();
            formatter.Serialize(stream, data.GetType());
            formatter.Serialize(stream, GetAsSerializable(data));

            // Now copy to an HGLOBAL
            byte[] bytes = stream.GetBuffer();
            IntPtr p = Marshal.AllocHGlobal(bytes.Length);
            try
            {
                Marshal.Copy(bytes, 0, p, bytes.Length);
            }
            catch
            {
                // Make sure to free the memory on exceptions
                Marshal.FreeHGlobal(p);
                throw;
            }

            // Now allocate an STGMEDIUM to wrap the HGLOBAL
            medium.unionmember = p;
            medium.tymed = ComTypes.TYMED.TYMED_HGLOBAL;
            medium.pUnkForRelease = null;
        }

I just create a MemoryStream, which I will serialize my object to, and then write an internally defined GUID to it. This GUID is a stamp for my code to recognize the data by when I go to get the data. I didn't mention it before, but the .NET GetData will only partially work for me. If a type is serializable (implements ISerializable or is marked by the SerializableAttribute) then the .NET GetData method will still work. If not, I'll have to use another new function, GetDataEx to get my data. We'll see that later. After the stamp, I serialize the type, which I'll need during deserialization because the actual data I serialize may be converted. Then, I use a helper method, GetAsSerializable, that will determine if the data's type is serializable, and if not, will try to convert it to a string using a type converter that the type may have declared through the TypeConverterAttribute. I won't show that code here, because it isn't directly relevant to drag and drop. Finally, I allocate an HGLOBAL and fill the STGMEDIUM structure before returning.

The SetDataEx function is an extension method on the .NET IDataObject interfaces:

        public static void SetDataEx(this IDataObject dataObject, string format, object data)
        {
            DataFormats.Format dataFormat = DataFormats.GetFormat(format);

            // Initialize the format structure
            ComTypes.FORMATETC formatETC = new ComTypes.FORMATETC();
            formatETC.cfFormat = (short)dataFormat.Id;
            formatETC.dwAspect = ComTypes.DVASPECT.DVASPECT_CONTENT;
            formatETC.lindex = -1;
            formatETC.ptd = IntPtr.Zero;

            // Try to discover the TYMED from the format and data
            ComTypes.TYMED tymed = GetCompatibleTymed(format, data);
            // If a TYMED was found, we can use the system DataObject
            // to convert our value for us.
            if (tymed != ComTypes.TYMED.TYMED_NULL)
            {
                formatETC.tymed = tymed;

                // Set data on an empty DataObject instance
                DataObject conv = new DataObject();
                conv.SetData(format, true, data);

                // Now retrieve the data, using the COM interface.
                // This will perform a managed to unmanaged conversion for us.
                ComTypes.STGMEDIUM medium;
                ((ComTypes.IDataObject)conv).GetData(ref formatETC, out medium);
                try
                {
                    // Now set the data on our data object
                    ((ComTypes.IDataObject)dataObject).SetData(ref formatETC, ref medium, true);
                }
                catch
                {
                    // On exceptions, release the medium
                    ReleaseStgMedium(ref medium);
                    throw;
                }
            }
            else
            {
                // Since we couldn't determine a TYMED, this data
                // is likely custom managed data, and won't be used
                // by unmanaged code, so we'll use our custom marshaling
                // implemented by our COM IDataObject extensions.

                ComTypes.ComDataObjectExtensions.SetManagedData((ComTypes.IDataObject)dataObject, format, data);
            }
        }

There is a little bit of magic here. You'll see that I call the SetManagedData function at the end of SetDataEx as a last resort only. The rest of the code actually uses a temporary .NET DataObject instance to act as a converter. First I fill a FORMATETC structure for my desired format. Then I attempt to determine the TYMED based on the format name and data. The method GetCompatibleType is uninteresting, and is based on reflection of the .NET DataObject conversion capabilities. ("Compatible" means that it can be converted during a COM GetData call to the .NET DataObject.) It is slightly different for SWF and WPF, but is largely the same. Once I determine a compatible type, I set the data on the temporary .NET DataObject using its .NET SetData method, then call the COM GetData method, which will do marshaling according to the format and type of data. Then, since the marshaling is done, I can just set the data onto the real data object through the COM SetData method. Notice, only if I fail to determine a compatible TYMED do I do my own marshaling using SetManagedData.

Managed SetData Summary

Now that SetDataEx is in place, it can be used like the .NET SetData method. The only catch is that the data must be either serializable, or have an associated TypeConverter that supports conversion to/from a string. The GetData function will still work other than when the data's type is not serializable. In that case, you can call GetDataEx, which I won't show here. It basically confirms the stamp, then deserializes the data's type, then the data, and performs conversion through a TypeConverter if necessary.

The DragSourceHelper

I've discussed a lot about the heart of the problem and have provided a solid foundation to build a complete solution. However, there are some intricacies to proper implementation that I haven't addressed. We need to handle, as a drag source, the cases when the cursor is over a non-drag image enabled drop target, a drop target that doesn't set the drop description, and the Windows Taskbar (which just throws a curve ball for fun). To handle all this, I've implemented the DragSourceHelper, which is a static class that keeps some smart context to eliminate much of the repetitive client coding and to help you manage drag and drop with less hassle.

Default QueryContinueDrag Handler

I'll be honest up front. The QueryContinueDrag event seems pointless to me. I don't know when a drag source should ever have control over whether to continue a drag and drop operation. Having said that, there is one thing I could think of, that the system, for whatever reason, doesn't handle. That is the escape key. If the user presses escape during the drag and drop operation, most drag sources will just cancel the operation (or should, anyway). That is what my default handler does:

        public static void DefaultQueryContinueDrag(QueryContinueDragEventArgs e)
        {
            if (e.EscapePressed)
            {
                e.Action = DragAction.Cancel;
            }
        }

Not much more to say here. This method belongs to my DragSourceHelper class and can be called directly from your QueryContinueDrag event handler, or I also expose a QueryContinueDragEventHandler method, so that you can assign it directly as the handler of your QueryContinueDrag event:

        public static void DefaultQueryContinueDragHandler(object sender, QueryContinueDragEventArgs e)
        {
            DefaultQueryContinueDrag(e);
        }

There isn't anything more to say here. For my implementation, that is about all there is.

Default GiveFeedback Handler

The GiveFeedback event handler has a lot more to do than the QueryContinueDrag handler. This is because I have several cases I have to cover:

• Drop targets that don't support drag images

Problem: If I assume all drop targets support drag images and don't make a special case, I'll end up showing them a plain arrow cursor, and the drop action will be completely unknown to the user.

Solution: For drop targets that don't support drag images, the drag image manager provides a piece of data in my data object that let's me know this is the case. The "IsShowingLayered" data format is a boolean value, true to indicate that the drop target is showing the drag image, false to indicate not. If the flag is on, I'll set UseDefaultCursors to false, then explicitly set the arrow cursor as my current mouse cursor. If it is not on, I'll set UseDefaultCursors to true, and I won't set the current cursor explicitly.

• Drop targets that support drag images but don't set the drop description

Problem: These targets talk to the Shell's IDropTargetHelper implementation, but don't set the drop description. For these targets, we can provide a default drop description instead of the out-of-date default drag and drop cursors.

Solution: I can handle this by taking care to handle all drag and drop events properly. For one, during DragLeave, I can make sure to set my drop description as invalid, by using the DropImageType.Invalid type. That is what Windows Explorer does, too, so we are consistent. Next, through GiveFeedback, I can detect this invalid drop description, then look at the current drop effect, set by the drop target, and set a default description. That part is easy, the only thing you have to really worry about is the fact that drop targets that don't set the drop description won't invalidate them either! That means I need to track when I have a default drop description currently set. Then I can detect changes to the drop effect and change the current drop description. I'll do this with a flag that is associated to the IDataObject. Well, I lied. That isn't the only thing to worry about. You also need to detect when, after you set a default drop description, a drop target sets a drop description. This can be caused by the mouse leaving the drop target that doesn't set the drop description and entering a drop target that does. I detect this by implementing an IAdviseSink and listening to DataChanged events on the DataObject. I'll address that in the next section, Adding Support for AdviseSinks to the DataObject.

• The Windows Taskbar

Problem: The Taskbar supports drag images, but doesn't set the drop description. The reason it is in a different category is because it is the most prominent place you'll notice that when you drag over it, from Windows Explorer, the no-smoking icon appears (drop effect None) but you won't see drag text. This is actually true for any drop target that supports drag images, but doesn't set a drop description and has a drop effect of None.

Solution: I can implement this by simply not invalidating the drag image after it has been rendered. That is, if I don't invalidate the drag image, then the drop description's icon will not refresh with the text after a pause. In order to implement this properly, I have to invalidate the drag image once after the drop description is set to None, and then not until it changes again.

The handler, after all of these cases are covered, looks like this:

        public static void DefaultGiveFeedback(IDataObject data, GiveFeedbackEventArgs e)
        {
            // For drop targets that don't set the drop description, we'll
            // set a default one. Drop targets that do set drop descriptions
            // should set an invalid drop description during DragLeave.
            bool setDefaultDropDesc = false;
            bool isDefaultDropDesc = IsDropDescriptionDefault(data);
            DropImageType currentType = DropImageType.Invalid;
            if (!IsDropDescriptionValid(data) || isDefaultDropDesc)
            {
                currentType = GetDropImageType(data);
                setDefaultDropDesc = true;
            }

            if (IsShowingLayered(data))
            {
                // The default drag source implementation uses drop descriptions,
                // so we won't use default cursors.
                e.UseDefaultCursors = false;
                Cursor.Current = Cursors.Arrow;
            }
            else
                e.UseDefaultCursors = true;

            // We need to invalidate the drag image to refresh the drop description.
            // This is tricky to implement correctly, but we try to mimic the Windows
            // Explorer behavior. We internally use a flag to tell us to invalidate
            // the drag image, so if that is set, we'll invalidate. Otherwise, we
            // always invalidate if the drop description was set by the drop target,
            // *or* if the current drop image is not None. So if we set a default
            // drop description to anything but None, we'll always invalidate.
            if (InvalidateRequired(data) || !isDefaultDropDesc || currentType != DropImageType.None)
            {
                InvalidateDragImage(data);

                // The invalidate required flag only lasts for one invalidation
                SetInvalidateRequired(data, false);
            }

            // If the drop description is currently invalid, or if it is a default
            // drop description already, we should check about re-setting it.
            if (setDefaultDropDesc)
            {
                // Only change if the effect changed
                if ((DropImageType)e.Effect != currentType)
                {
                    if (e.Effect == DragDropEffects.Copy)
                        data.SetDropDescription(DropImageType.Copy, "Copy", "");
                    else if (e.Effect == DragDropEffects.Link)
                        data.SetDropDescription(DropImageType.Link, "Link", "");
                    else if (e.Effect == DragDropEffects.Move)
                        data.SetDropDescription(DropImageType.Move, "Move", "");
                    else if (e.Effect == DragDropEffects.None)
                        data.SetDropDescription(DropImageType.None, null, null);
                    SetDropDescriptionIsDefault(data, true);

                    // We can't invalidate now, because the drag image manager won't
                    // pick it up... so we set this flag to invalidate on the next
                    // GiveFeedback event.
                    SetInvalidateRequired(data, true);
                }
            }
        }

You'll notice that I first detect if I will need to set the default description, and get the current drop image type. I don't set the default drop description at this point, because it doesn't make a difference. Calling InvalidateDragImage after setting the drop description here will not truly invalidate the drag image. Next, I determine if the drop target supports drag images via my IsShowingLayered helper function. If so, we won't use default cursors, else we will. Next, we determine if we should invalidate the drag image. Basically, if our flag is set from a previous call, or if the drop description was manually set by the drop target, or the drop description's drop image type is not None, then we'll invalidate, plus clear the flag to invalidate next time. Notice, that only when the None effect is applied do we possibly not invalidate the drag image.

Adding Support for AdviseSinks to the DataObject

In order for the GiveFeedback handler described above to work as expected, we have to detect when a drop target manually sets a drop description. There are a couple of ways to do this, but the most reliable is to listen to DataChanged events from the IDataObject. The COM IDataObject has a couple of methods that, until now, have been left as unsupported. They are DAdvise and DUnadvise. They are simply methods that enable you to pass an instance of IAdviseSink to listen for data events. The IAdviseSink interface covers advisory tasks for several system components, but for drag and drop, we just need to implement the OnDataChanged method:

        private class AdviseSink : ComTypes.IAdviseSink
        {
            // The associated data object
            private IDataObject data;

            /// <summary>
            /// Creates an AdviseSink associated to the specified data object.
            /// </summary>
            /// <param name="data">The data object.</param>
            public AdviseSink(IDataObject data)
            {
                this.data = data;
            }

            /// <summary>
            /// Handles DataChanged events from a COM IDataObject.
            /// </summary>
            /// <param name="format">The data format that had a change.</param>
            /// <param name="stgmedium">The data value.</param>
            public void OnDataChange(ref ComTypes.FORMATETC format, ref ComTypes.STGMEDIUM stgmedium)
            {
                // We listen to DropDescription changes, so that we can unset the IsDefault
                // drop description flag.
                object odd = ComTypes.ComDataObjectExtensions.GetDropDescription((ComTypes.IDataObject)data);
                if (odd != null)
                    DragSourceHelper.SetDropDescriptionIsDefault(data, false);
            }

            #region Unsupported callbacks

            public void OnClose()
            {
                throw new NotImplementedException();
            }

            public void OnRename(System.Runtime.InteropServices.ComTypes.IMoniker moniker)
            {
                throw new NotImplementedException();
            }

            public void OnSave()
            {
                throw new NotImplementedException();
            }

            public void OnViewChange(int aspect, int index)
            {
                throw new NotImplementedException();
            }

            #endregion // Unsupported callbacks
        }

Because we only care about DropDescription changes, we will connect an instance of the AdviseSink class for notifications of that format only. The OnDataChange function is the callback from the IDataObject and simply verifies the drop description is accessible and then unsets the IsDefault flag for the data object. This tells the DragSourceHelper that this data object no longer has the default drop description.

The drop description flags I've mentioned, IsDefault and InvalidateRequired, are just a couple of internal flags, which I store in a Dictionary keyed on the IDataObject itself. That way I don't need to set internal-only data on the data object.

The Do-It-All Functions

The DragSourceHelper class isn't too complex, but it is more than I want to go into here. Most of it is just convenient do-it-all or do-it-in-less-lines-of-code functions. Having said that, there are a couple of functions I want to point out. You'll see in the examples, I use the DragSourceHelper.DoDragDrop function. The DoDragDrop function is a one-liner to call for the drag source, and handles all the internal drag and drop logic from the drag source side. The only thing you need to do is prepare a drag image, call DoDragDrop, and then handle the drag drop effect. Of course, for the cases that you need a little more granularity, there are several available functions and overrides to help you get what you need to get done without the hassle. Let's take a look at the DoDragDrop function (I included source for a couple of the called functions for reference):

        public static DragDropEffects DoDragDrop(
            Control control,
            System.Drawing.Point cursorOffset,
            DragDropEffects allowedEffects,
            params KeyValuePair<string, object>[] data)
        {
            IDataObject dataObject = RegisterDefaultDragSource(control, cursorOffset);
            return DoDragDropInternal(control, dataObject, allowedEffects, data);
        }

        public static IDataObject RegisterDefaultDragSource(
            Control control, System.Drawing.Point cursorOffset)
        {
            IDataObject data = CreateDataObject(control, cursorOffset);
            RegisterDefaultDragSource(control, data);
            return data;
        }

        public static void RegisterDefaultDragSource(Control control, IDataObject data)
        {
            // Cache the drag source and the associated data object
            DragSourceEntry entry = new DragSourceEntry(data);
            if (!s_dataContext.ContainsKey(control))
                s_dataContext.Add(control, entry);
            else
                s_dataContext[control] = entry;

            // We need to listen for drop description changes. If a drop target
            // changes the drop description, we shouldn't provide a default one.
            entry.adviseConnection = ComTypes.ComDataObjectExtensions.Advise(
                (ComTypes.IDataObject)data, new AdviseSink(data), DropDescriptionFormat, 0);

            // Hook up the default drag source event handlers
            control.GiveFeedback += new GiveFeedbackEventHandler(DefaultGiveFeedbackHandler);
            control.QueryContinueDrag += new QueryContinueDragEventHandler(DefaultQueryContinueDragHandler);
        }

        private static DragDropEffects DoDragDropInternal(
            Control control, IDataObject dataObject,
            DragDropEffects allowedEffects,
            KeyValuePair<string, object>[] data)
        {
            // Set the data onto the data object.
            if (data != null)
            {
                foreach (KeyValuePair<string, object> dataPair in data)
                    dataObject.SetDataEx(dataPair.Key, dataPair.Value);
            }

            try
            {
                return control.DoDragDrop(dataObject, allowedEffects);
            }
            finally
            {
                UnregisterDefaultDragSource(control);
            }
        }

What you'll see is that the DoDragDrop creates an instance of the DataObject, including registering it internally for default GiveFeedback and QueryContinueDrag event handlers and DataChange handler on the DropDescription format. It then adds any data you passed in, and calls the .NET DoDragDop, and then unregisters the event handlers before it returns the result drag drop effect.

This reduces the client code to a one-liner, as in the examples:

        DragSourceHelper.DoDragDrop(button1, new Point(e.X, e.Y), DragDropEffects.Link | DragDropEffects.Copy,
            new KeyValuePair<string, object>(DataFormats.Text, "Hello Drag and Drop!"),
            new KeyValuePair<string, object>(DataFormats.Html, "<h4>Hello Drag and Drop!</h4>"));

In this example, you may have noticed, we are initializing the drag image from the button control itself, instead of a custom bitmap. Other overrides exist to help you achieve your most intriguing drag and drop experience, yet.

Conclusion

The last couple of weeks have gone deep into the implementation of a complete Shell-integrated drag and drop experience for your .NET apps. Whether you are working in WinForms or WPF, you can provide great user feedback during drag and drop operations. With WPF, this provides a great base for applications that are highly dynamic in the user experience. In WinForms, you can get that true integration that you've felt like .NET lacked.

I encourage you to use the source provided, although please keep in mind it is Microsoft copyrighted.

Source Code

All source for this article is provided below.

This is part of a 3 part series:

1. Shell Style Drag and Drop in .NET (WPF and WinForms)
2. Shell Style Drag and Drop in .NET - Part 2
3. Shell Style Drag and Drop in .NET - Part 3

Introduction

Last week, in Shell Style Drag and Drop in .NET (WPF and WinForms), I looked at hooking up the COM interfaces necessary to implement drag images using the Windows Shell. This week, I'm going to introduce some .NET 3.5 extensions (which can be easily converted into .NET 3.0 and earlier static helper functions) that reduce the code overhead of implementing great drag and drop preview images in your .NET applications.

NOTE: Like last week, this is not an in depth overview of drag and drop in .NET, but instead focuses on utilizing the Shell's drag image manager to develop more tightly integrated drag and drop applications.

Background

For those of you who didn't read last week's post (or found it too long), here is a recap of the important COM interfaces:

• IDragSourceHelper - Exposed by the Shell to allow an application to specify the image that will be displayed during a Shell drag-and-drop operation.
• IDropTargetHelper - Exposes methods that allow drop targets to display a drag image while the image is over the target window.
• IDataObject - Specifies methods that enable data transfer and notification of changes in data (we only use it for the former).

I implemented IDataObject (the COM version, not System.Windows.IDataObject or System.Windows.Forms.IDataObject) and then imported the other interfaces, along with the DragDropHelper class, CoCreated using CLSID_DragDropHelper from the Shell.

The Extensions

Unlike last week, I won't take you step by step into the implementation of these extensions. But I will give you an overview of what I've added.

IDataObject Extensions

For each System.Windows.IDataObject and System.Windows.Forms.IDataObject, I added several extensions to simplify the initialization of the drag image. Since they are very similar, I'll use System.Windows.Forms as my example, and leave the WPF implementation to be explored by you.

Last week, I used code like this to start the drag and drop operation:

    void bt_MouseDown(object sender, MouseEventArgs e)
    {
        Bitmap bmp = new Bitmap(100, 100, PixelFormat.Format32bppArgb);
        using (Graphics g = Graphics.FromImage(bmp))
        {
            g.Clear(Color.Magenta);
            using (Pen pen = new Pen(Color.Blue, 4f))
            {
                g.DrawEllipse(pen, 20, 20, 60, 60);
            }
        }
        
        DataObject data = new DataObject(new DragDropLib.DataObject());
        
        ShDragImage shdi = new ShDragImage();
        Win32Size size;
        size.cx = bmp.Width;
        size.cy = bmp.Height;
        shdi.sizeDragImage = size;
        Point p = e.Location;
        Win32Point wpt;
        wpt.x = p.X;
        wpt.y = p.Y;
        shdi.ptOffset = wpt;
        shdi.hbmpDragImage = bmp.GetHbitmap();
        shdi.crColorKey = Color.Magenta.ToArgb();
        
        IDragSourceHelper sourceHelper = (IDragSourceHelper)new DragDropHelper();
        sourceHelper.InitializeFromBitmap(ref shdi, data);
        
        DoDragDrop(data, DragDropEffects.Copy);
    }

The overhead of creating and populating the ShDragImage structure can get tedious and dirties the code. Also, if you read the code last week, you may have noticed the usage of the DragDropHelper was always two lines, such as above. One line to create the object and cast to the interface, and one line to call into it. Granted, you could convert this into a one-liner, but it would be ugly, and still more typing than necessary. Because the InitializeFromBitmap method works on an existing IDataObject instance, I decided an extension was appropriate. Basically, the ShDragImage structure has two important pieces of information. The HBITMAP pointer, which indicates our drag image, and the offset of the cursor relative to the image. The size of the bitmap is certainly important, but that information can be derived from the Bitmap instance, so my extension ended up like this:

        /// <summary>
        /// Sets the drag image.
        /// </summary>
        /// <param name="dataObject">The DataObject to set the drag image on.</param>
        /// <param name="image">The drag image.</param>
        /// <param name="cursorOffset">The location of the cursor relative to the image.</param>
        public static void SetDragImage(this IDataObject dataObject, Image image, Point cursorOffset)
        {
            ShDragImage shdi = new ShDragImage();

            Win32Size size;
            size.cx = image.Width;
            size.cy = image.Height;
            shdi.sizeDragImage = size;

            Win32Point wpt;
            wpt.x = cursorOffset.X;
            wpt.y = cursorOffset.Y;
            shdi.ptOffset = wpt;

            shdi.crColorKey = Color.Magenta.ToArgb();

            // This HBITMAP will be managed by the DragDropHelper
            // as soon as we pass it to InitializeFromBitmap. If we fail
            // to make the hand off, we'll delete it to prevent a mem leak.
            IntPtr hbmp = GetHbitmapFromImage(image);
            shdi.hbmpDragImage = hbmp;

            try
            {
                IDragSourceHelper sourceHelper = (IDragSourceHelper)new DragDropHelper();

                try
                {
                    sourceHelper.InitializeFromBitmap(ref shdi, (ComIDataObject)dataObject);
                }
                catch (NotImplementedException ex)
                {
                    throw new Exception("A NotImplementedException was caught. "
                       + "This could be because you forgot to construct your DataObject "
                       + "using a DragDropLib.DataObject", ex);
                }
            }
            catch
            {
                DeleteObject(hbmp);
            }
        }

        /// <summary>
        /// Gets an HBITMAP from any image.
        /// </summary>
        /// <param name="image">The image to get an HBITMAP from.</param>
        /// <returns>An HBITMAP pointer.</returns>
        /// <remarks>
        /// The caller is responsible to call DeleteObject on the HBITMAP.
        /// </remarks>
        private static IntPtr GetHbitmapFromImage(Image image)
        {
            if (image is Bitmap)
            {
                return ((Bitmap)image).GetHbitmap();
            }
            else
            {
                Bitmap bmp = new Bitmap(image);
                return bmp.GetHbitmap();
            }
        }

You'll notice that the SetDragImage extension function simply takes an IDataObject (in this case it is the System.Windows.Forms.IDataObject), an Image, and a cursor offset. From these pieces of information, we can easily populate the ShDragImage struct and call the IDragSourceHelper.InitializeFromBitmap method.

With this extension, which I placed in the System.Windows.Forms namespace for convenience, the code to begin a drag and drop routine is much simpler:

    void bt_MouseDown(object sender, MouseEventArgs e)
    {
        Bitmap bmp = new Bitmap(100, 100, PixelFormat.Format32bppArgb);
        using (Graphics g = Graphics.FromImage(bmp))
        {
            g.Clear(Color.Magenta);
            using (Pen pen = new Pen(Color.Blue, 4f))
            {
                g.DrawEllipse(pen, 20, 20, 60, 60);
            }
        }
        
        DataObject data = new DataObject(new DragDropLib.DataObject());
        data.SetDragImage(bmp, e.Location);
        
        DoDragDrop(data, DragDropEffects.Copy);
    }

Ignoring the creation of the Bitmap, I can now start drag and drop using a drag image in 3 lines of code.

IDropTargetHelper Extensions

Much like IDataObject, I can also extend IDropTargetHelper. However, now, instead of extending System.Windows.IDataObject and System.Windows.Forms.IDataObject separately, we extend IDropTargetHelper with methods for both WPF and WinForms. Again, I'll work with the WinForms code, but the WPF extensions are very similar.

Last week, we used code that handled the drag and drop events like this:

    protected override void OnDragEnter(DragEventArgs e)
    {
        e.Effect = DragDropEffects.Copy;
        Point p = Cursor.Position;
        Win32Point wp;
        wp.x = p.X;
        wp.y = p.Y;
        IDropTargetHelper dropHelper = (IDropTargetHelper)new DragDropHelper();
        dropHelper.DragEnter(IntPtr.Zero, (ComIDataObject)e.Data, ref wp, (int)e.Effect);
    }

The code is simple enough, but again, it gets tedious to declare the Win32Point struct and populate it. So, we can implement some extensions that use the native System.Windows.Forms and System.Drawing types:

        /// <summary>
        /// Notifies the DragDropHelper that the specified Control received
        /// a DragEnter event.
        /// </summary>
        /// <param name="dropHelper">The DragDropHelper instance to notify.</param>
        /// <param name="control">The Control the received the DragEnter event.</param>
        /// <param name="data">The DataObject containing a drag image.</param>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void DragEnter(this IDropTargetHelper dropHelper, Control control,
            IDataObject data, Point cursorOffset, DragDropEffects effect)
        {
            IntPtr controlHandle = IntPtr.Zero;
            if (control != null)
                controlHandle = control.Handle;
            Win32Point pt = SwfDragDropLibExtensions.ToWin32Point(cursorOffset);
            dropHelper.DragEnter(controlHandle, (ComIDataObject)data, ref pt, (int)effect);
        }

        /// <summary>
        /// Notifies the DragDropHelper that the current Control received
        /// a DragOver event.
        /// </summary>
        /// <param name="dropHelper">The DragDropHelper instance to notify.</param>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void DragOver(this IDropTargetHelper dropHelper, Point cursorOffset, DragDropEffects effect)
        {
            Win32Point pt = SwfDragDropLibExtensions.ToWin32Point(cursorOffset);
            dropHelper.DragOver(ref pt, (int)effect);
        }

        /// <summary>
        /// Notifies the DragDropHelper that the current Control received
        /// a Drop event.
        /// </summary>
        /// <param name="dropHelper">The DragDropHelper instance to notify.</param>
        /// <param name="data">The DataObject containing a drag image.</param>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void Drop(this IDropTargetHelper dropHelper, IDataObject data, Point cursorOffset, DragDropEffects effect)
        {
            Win32Point pt = SwfDragDropLibExtensions.ToWin32Point(cursorOffset);
            dropHelper.Drop((ComIDataObject)data, ref pt, (int)effect);
        }

On top of that, I wanted to add one more layer. Since I always need an instance of the DragDropHelper, why not wrap that into some static methods? What I ended up doing is creating a static class called DropTargetHelper. This class maintains a static instance of the DragDropHelper and exposes some static methods. I also put this class in the System.Windows.Forms namespace, since that is generally where it would be used. Note that these methods don't do anything except add a layer of abstraction around the creation of the DragDropHelper class:

namespace System.Windows.Forms
{
    public static class DropTargetHelper
    {
        /// <summary>
        /// Internal instance of the DragDropHelper.
        /// </summary>
        private static IDropTargetHelper s_instance = (IDropTargetHelper)new DragDropHelper();

        static DropTargetHelper()
        {
        }

        /// <summary>
        /// Notifies the DragDropHelper that the specified Control received
        /// a DragEnter event.
        /// </summary>
        /// <param name="control">The Control the received the DragEnter event.</param>
        /// <param name="data">The DataObject containing a drag image.</param>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void DragEnter(Control control, IDataObject data, Point cursorOffset, DragDropEffects effect)
        {
            SwfDropTargetHelperExtensions.DragEnter(s_instance, control, data, cursorOffset, effect);
        }

        /// <summary>
        /// Notifies the DragDropHelper that the current Control received
        /// a DragOver event.
        /// </summary>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void DragOver(Point cursorOffset, DragDropEffects effect)
        {
            SwfDropTargetHelperExtensions.DragOver(s_instance, cursorOffset, effect);
        }

        /// <summary>
        /// Notifies the DragDropHelper that the current Control received
        /// a DragLeave event.
        /// </summary>
        public static void DragLeave()
        {
            s_instance.DragLeave();
        }

        /// <summary>
        /// Notifies the DragDropHelper that the current Control received
        /// a DragOver event.
        /// </summary>
        /// <param name="data">The DataObject containing a drag image.</param>
        /// <param name="cursorOffset">The current cursor's offset relative to the window.</param>
        /// <param name="effect">The accepted drag drop effect.</param>
        public static void Drop(IDataObject data, Point cursorOffset, DragDropEffects effect)
        {
            SwfDropTargetHelperExtensions.Drop(s_instance, data, cursorOffset, effect);
        }

        /// <summary>
        /// Tells the DragDropHelper to show or hide the drag image.
        /// </summary>
        /// <param name="show">True to show the image. False to hide it.</param>
        public static void Show(bool show)
        {
            s_instance.Show(show);
        }
    }
}

NOTE: You may notice I'm calling my extension methods as static methods. This is because my code can be easily converted to work with .NET 3.0 or before by removing (or compiling out by way of preprocessor directives) the "this" keyword in front of the first parameter of the extension methods. I was going to include source that was compatible with .NET 3.0 but decided against it last minute. Anyway, the extensions, whether called as a static method or as an extension method, will do the same thing.

Now my drag event handlers can be simplified to something like this:

    protected override void OnDragEnter(DragEventArgs e)
    {
        e.Effect = DragDropEffects.Copy;
        DropTargetHelper.DragEnter(this, e.Data, Cursor.Position, e.Effect);
    }

Much simpler, and much less tedious to implement across 50 Forms in some enterprise application.

Conclusion

Drag and drop is a powerful tool for the end user experience. With a little work, we can implement a set of drag and drop helpers that keep our code tidy and add a great user experience for a minimal cost.

Source Code

The sources are available here:

What's Next?

Implementing the fundamentals is all good, but nothing feels better than putting it to use. In my next post, we'll look at an example of really putting this technology to use in the end user experience.