Windows with C++ - Windows Runtime Components with MIDL

By Kenny Kerr | August 2015

Kenny Kerr In my July 2015 column (msdn.microsoft.com/magazine/mt238401), I introduced the concept of Windows Runtime (WinRT) components as the evolution of the COM programming paradigm. Whereas Win32 put COM on the side, the Windows Runtime puts COM front and center. The Windows Runtime is the successor to Win32, the latter being an umbrella term for the Windows API as it encompasses many disparate technologies and programming models. The Windows Runtime provides a consistent and unified programming model, but in order for it to succeed developers both inside and outside of Microsoft need better tools to develop WinRT components and to use those components from within apps.

The primary tool provided by the Windows SDK to serve this need is the MIDL compiler. In the July column, I showed how the MIDL compiler could produce the Windows Runtime Metadata (WINMD) file most language projections require to consume WinRT components. Of course, any longtime developer on the Windows platform will be aware that the MIDL compiler also produces code that a C or C++ compiler can consume directly. In fact, MIDL itself knows nothing about the WINMD file format. It’s principally about parsing IDL files and producing code for C and C++ compilers in order to support COM and remote procedure call (RPC) development and the production of proxy DLLs. The MIDL compiler is such a historically critical piece of machinery that the engineers who developed the Windows Runtime chose not to take the risk of breaking it and instead developed a “sub compiler” responsible only for the Windows Runtime. Developers are not usually aware of this subterfuge—and need not be—but it helps explain the way the MIDL compiler works in practice.

Let’s look at some IDL source code and see what’s really going on with the MIDL compiler. Here’s an IDL source file that defines a classic COM interface:

C:\Sample>type Sample.idl
import "unknwn.idl";
[uuid(e21df825-937d-4b0b-862e-e411b57e280e)]
interface IHen : IUnknown
{
  HRESULT Cluck();
}

Classic COM doesn’t have a strong notion of namespaces so the IHen interface is simply defined at file scope. The definition of IUnknown must also be imported prior to use. I can then just pass this file through the MIDL compiler to produce a number of artifacts:

C:\Sample>midl Sample.idl
C:\Sample>dir /b
dlldata.c
Sample.h
Sample.idl
Sample_i.c
Sample_p.c

The dlldata.c source file contains a few macros that implement the necessary exports for a proxy DLL. The Sample_i.c contains the IHen interface’s GUID, should you be using a 25-year-old compiler that lacks support for the uuid __declspec that attaches GUIDs to types. Then there’s Sample_p.c, which contains the marshaling instructions for the proxy DLL. I’ll gloss over these for the moment and instead focus on Sample.h, which contains something quite handy. If you overlook all of the horrible macros intended to help C developers use COM (the horror!) you’ll find this:

MIDL_INTERFACE("e21df825-937d-4b0b-862e-e411b57e280e")
IHen : public IUnknown
{
public:
  virtual HRESULT STDMETHODCALLTYPE Cluck( void) = 0;
};

It’s not elegant C++ but, after preprocessing, it amounts to a C++ class that inherits from IUnknown and adds a pure virtual function of its own. This is handy because it means you don’t have to write this by hand, possibly introducing a mismatch between the C++ definition of the interface and the original IDL definition that other tools and languages might consume. So that’s the essence of what the MIDL compiler provides for C++ developers, producing a translation of the IDL source code in such a way that a C++ compiler can consume those types directly.

Now let’s return to the Windows Runtime. I’ll update the IDL source code just slightly to comply with the more stringent requirements for WinRT types:

C:\Sample>type Sample.idl
import "inspectable.idl";
namespace Sample
{
  [uuid(e21df825-937d-4b0b-862e-e411b57e280e)]
  [version(1)]
  interface IHen : IInspectable
  {
    HRESULT Cluck();
  }
}

WinRT interfaces must inherit directly from IInspectable, and a namespace is used in part to associate the types with the implementing component. If I try to compile it as before, I run into a problem:

.\Sample.idl(3) : error MIDL2025 : syntax error : expecting an interface name or DispatchInterfaceName or CoclassName or ModuleName or LibraryName or ContractName or a type specification near "namespace"

The MIDL compiler doesn’t recognize the namespace keyword and gives up. That’s what the /winrt command-line option is for. It tells the MIDL compiler to pass the command line directly to the MIDLRT compiler to preprocess the IDL source file. It is this second compiler—MIDLRT—that expects the /metadata_dir command-line option I mentioned in the July column:

C:\Sample>midl /winrt Sample.idl /metadata_dir
  "C:\Program Files (x86)\Windows Kits ..."

As further evidence of this, take a closer look at the MIDL compiler output and you’ll see what I mean:

C:\Sample>midl /winrt Sample.idl /metadata_dir "..."
Microsoft (R) 32b/64b MIDLRT Compiler Engine Version 8.00.0168
Copyright (c) Microsoft Corporation. All rights reserved.
MIDLRT Processing .\Sample.idl
.
.
.
Microsoft (R) 32b/64b MIDL Compiler Version 8.00.0603
Copyright (c) Microsoft Corporation. All rights reserved.
Processing C:\Users\Kenny\AppData\Local\Temp\Sample.idl-34587aaa
.
.
.

I’ve removed some of the processing of dependencies in order to highlight the key points. Calling the MIDL executable with the /winrt options blindly passes the command line to the MIDLRT executable before exiting. MIDLRT parses the IDL first to generate the WINMD file, but it also produces another temporary IDL file. This temporary IDL file is a translation of the original with all of the WinRT-specific keywords, such as namespaces, replaced in such a way that the original MIDL compiler will accept it. MIDLRT then calls the MIDL executable again but without the /winrt option and with the location of the temporary IDL file so it can produce the original set of C and C++ headers and source files as before.

The namespace in the original IDL file is removed and the name of the IHen interface is decorated in the temporary IDL file as follows:

interface __x_Sample_CIHen : IInspectable
.
.
.

This is actually an encoded form of the type name that’s interpreted by the MIDL compiler given the /gen_namespace command-line option that MIDLRT uses when calling MIDL with the preprocessed output. The original MIDL compiler can then process this directly without specific knowledge of the Windows Runtime. This is just one example but it gives you an idea of how the new tooling makes the most of existing technology to get the job done. If you’re curious to see how this works, you might poke around the temporary folder the MIDL compiler traces out, only to find that those files—Sample.idl-34587aaa in the previous example—are missing. The MIDLRT executable is careful to clean up after itself, but if you include the /savePP command-line option, MIDL won’t delete these temporary preprocessor files. Anyway, a bit more preprocessing is thrown in and the resulting Sample.h now has something that even a C++ compiler will recognize as a namespace:

namespace Sample {
  MIDL_INTERFACE("e21df825-937d-4b0b-862e-e411b57e280e")
  IHen : public IInspectable
  {
  public:
    virtual HRESULT STDMETHODCALLTYPE Cluck( void) = 0;
  };
}

I can then implement this interface as before, confident that the compiler will pick up any discrepancies between my implementation and the original definitions that I coded in IDL. On the other hand, if you only need MIDL to produce the WINMD file and don’t need all of the source files for a C or C++ compiler, you can avoid all of the extra build artifacts with the /nomidl command-line option. This option is passed, along with all the rest, by the MIDL executable to the MIDLRT executable. MIDLRT then skips the last step of calling MIDL again after it finishes producing the WINMD file. It’s also customary, when using a Windows Runtime ABI produced by MIDL, to include the /ns_prefix command-line option so that resulting types and namespaces are enclosed within the “ABI” namespace as follows:

namespace ABI {
  namespace Sample {
    MIDL_INTERFACE("e21df825-937d-4b0b-862e-e411b57e280e")
    IHen : public IInspectable
    {
    public:
      virtual HRESULT STDMETHODCALLTYPE Cluck( void) = 0;
    };
  }
}

Finally, I should mention that neither MIDL nor MIDLRT is enough to produce a self-contained WINMD file that sufficiently describes a component’s types. If you happen to reference external types, usually other types defined by the OS, the WINMD file, produced by the process described thus far, must still be merged with the main metadata file for the version of Windows you’re targeting. Let me illustrate the problem.

I’ll start with an IDL namespace describing both an IHen interface and an activatable Hen class that implements this interface, as shown in Figure 1.

Figure 1 The Hen Class in IDL

namespace Sample
{
  [uuid(e21df825-937d-4b0b-862e-e411b57e280e)]
  [version(1)]
  interface IHen : IInspectable
  {
    HRESULT Cluck();
  }
  [version(1)]
  [activatable(1)]
  runtimeclass Hen
  {
    [default] interface IHen;
  }
}

I’ll then implement it using the same technique I described in the July column, except that now I can rely on the definition of IHen as provided by the MIDL compiler. Now, inside a WinRT app, I can simply create a Hen object and call the Cluck method. I’ll use C# to illustrate the app side of the equation:

public void SetWindow(CoreWindow window)   
{
  Sample.IHen hen = new Sample.Hen();
  hen.Cluck();
}

The SetWindow method is part of the IFrameworkView implementation provided by the C# app. (I described IFrameworkView in my August 2013 column, which you can read at msdn.microsoft.com/magazine/jj883951.) And, of course, this works. C# is utterly dependent on the WINMD metadata describing the component. On the other hand, it sure makes sharing native C++ code with C# clients a breeze. Most of the time, anyway. One problem that arises is if you reference external types, as I alluded to a moment ago. Let’s update the Cluck method to require a CoreWindow as an argument. CoreWindow is defined by the OS so I can’t simply define it inside my IDL source file.

First, I’ll update the IDL to take a dependency on the ICoreWindow interface. I’ll simply import the definition as follows:

import "windows.ui.core.idl";

And then I’ll add an ICoreWindow parameter to the Cluck method:

HRESULT Cluck([in] Windows.UI.Core.ICoreWindow * window);

The MIDL compiler will turn that “import” into a #include of windows.ui.core.h inside the header that it generates so all I need to do is update my Hen class implementation:

virtual HRESULT __stdcall Cluck(ABI::Windows::UI::Core::ICoreWindow *) 
  noexcept override
{
  return S_OK;
}

I can now compile the component as before and ship it to the app developer. The C# app developer dutifully updates the Cluck method call with a reference to the app’s CoreWindow as follows:

public void SetWindow(CoreWindow window)
{
  Sample.IHen hen = new Sample.Hen();
  hen.Cluck(window);
}

Unfortunately, the C# compiler now complains:

error CS0012: The type 'ICoreWindow' is defined in an assembly
  that is not referenced.

You see, the C# compiler doesn’t recognize the interfaces as being the same. The C# compiler isn’t satisfied with just a matching type name and can’t make the connection with the Windows type of the same name. Unlike C++, C# is very much dependent on binary type information to connect the dots. To solve this problem, I can employ another tool provided by the Windows SDK that will compose or merge the metadata from the Windows OS along with my component’s metadata, correctly resolving the ICoreWindow to the main metadata file for the OS. This tool is called MDMERGE:

c:\Sample>mdmerge /i . /o output /partial /metadata_dir "..."

The MIDLRT and MDMERGE executables are quite particular about their command-line arguments. You need to get them just right in order for it to work. In this case, I can’t simply update Sample.winmd in place by pointing the /i (input) and /o (output) options to the same folder because MDMERGE actually deletes the input WINMD file upon completion. The /partial option tells MDMERGE to look for the unresolved ICoreWindow interface in the metadata provided by the /metadata_dir option. This is called the reference metadata. MDMERGE can thus be used to merge multiple WINMD files but in this case, I’m just using it to resolve references for OS types.

At this point, the resulting Sample.winmd correctly points to the metadata from the Windows OS when referring to the ICoreWindow interface and the C# compiler is satisfied and will compile the app as written. Join me next month as I continue to explore the Windows Runtime from C++.

Kenny Kerr is a computer programmer based in Canada, as well as an author for Pluralsight and a Microsoft MVP. He blogs at kennykerr.ca and you can follow him on Twitter at twitter.com/kennykerr.

Thanks to the following Microsoft technical expert for reviewing this article: Larry Osterman

Windows with C++ - Windows Runtime Components with MIDL

Additional resources