I'm proud to say that today the OneNote API went live, with our dev site at https://dev.onenote.com, blog at https://blogs.msdn.com/onenotedev, twitter @onenotedev, samples https://www.github.com/onenotedev

About six months ago, I left the wonderful Visual Studio team, to become API architect for the brand new OneNote service.  This was a bit of a wrench after ten amazing years in VS, but I realized that I wanted to take advantage of the opportunity working at Microsoft affords to experience really diverse businesses, so here I am, and I am loving it!  The new team has been amazingly friendly and welcoming and we're about as excited as it is possible to be today to get out into the public sphere and start getting broader feedback.  You should have seen the ship room packed with people in scary purple wigs at about 7am this morning watching us flip the switches to go live with bated breath.

Some come one, come all and tell us what you think, what you want to see next and above all, what cool things you're doing with our API to help people capture their digital memories.

1. Enable fine-grained template reuse with a new ‘once’ option on the <#@ include #> directive.

This feature makes it really easy to build up a library of reusable T4 snippets that you can include at will without worrying that some other snippet has already included them.  For example, I’ve now got a library of very fine-grained snippets that deal with template processing and C# generation.  In turn, these are used by some more task-specific utilities such as generating exceptions, which I can then use from any more application-specific template.  You can see in this dependency graph that my indentation helpers are used all over the place.  Now I don’t have to worry about clashes.

Customers ask me a lot about strategies for building reusable template libraries and the ‘once’ feature is the last major unblocker to a smooth experience for big libraries.  This feature works regardless of the host in use.  Here’s the syntax:

<#@ include file=“myinterestingsnippet.t4” once=”true” #>

2. Enable project-relative assembly references and includes across IDE and msbuild.

Until now, it’s been awkward to create templates that reference assemblies from standard project-relative directories and then transform them in both the IDE and from msbuild.  Typically it required environment variables, which most people hate!  We’ve fixed that with this feature.  You can now use project properties with include and assembly directives.  This feature works in the standard IDE host and in the msbuild host.  Here’s an example – imagine we have some helper code in an assembly called myLib.dll stored in a standard ‘libs’ directory.  Something magnificent, like this:

namespace MyLib

{

    public static class MyClass

    {

        public static string MyMethod()

        {

            return "World";

        }

    }

}

Now let’s use that helper from a template – in the IDE we could use macros like $(SolutionDir), but those don’t work in msbuild, so we’ll use a project variable instead:

<#@ template debug="false" hostspecific="false" language="C#" #>

<#@ assembly name="$(libDir)\MyLib.dll" #>

<#@ output extension=".txt" #>

Hello <#= MyLib.MyClass.MyMethod() #>

Of course, we’ll need to define the variable in our project file by adding a snippet like this:

<PropertyGroup>

    <libDir>$(MSBuildProjectDirectory)\..\libs</libDir>

</PropertyGroup>

At this point, the template should work in the IDE, but we need to take one more step before it works in msbuild.  We need to let the T4 msbuild task know which variables we’re using.  We add one final snippet to do that:  

  <ItemGroup>

    <T4ParameterValues Include="libDir">

      <Value>$(libDir)</Value>

    </T4ParameterValues>

  </ItemGroup>

3. Standard host used in ASP.Net scaffolding

This isn’t strictly a T4 feature, but it’s great news.  The very cool new ASP.Net scaffolding feature uses the standard VS T4 host, so you can use the full power of T4, and any include libraries you have when building scaffolding templates.

4. Tidy up after very spendy templates with CleanupBehavior directive.

Sometimes templates get big.  Or crufty. Or both. Or they load a bajillion assemblies to do their work.  This can bloat up your IDE’s memory footprint or lock assemblies in memory.  Now there’s a great big hammer to solve this problem.  Add the following line to your template and the appdomain it runs in will get blown away after every run. 

<#@ CleanupBehavior processor="T4VSHost" CleanupAfterProcessingtemplate="true" #>

This defeat’s T4’s normal caching behavior, so it slows things down a bit, but you get a spangly clean IDE every time.  This one only runs in the VS IDE host.

5. A bunch of small bugs, but that goes without saying.

Enjoy the release.

While previously a process for debugging a misbehaving template existed, it was not intuitive and somewhat of a hassle (requiring multiple instances of Visual Studio). Our intent was to remedy this situation and we approached it with two primary goals in mind

1. make the ability to debug a template easily accessible and
2. reproduce template behavior identical to that of the template running as usual in Visual Studio.

Unfortunately, there are cases where those goals are at odds with one another. T4 templates currently run within the Visual Studio process confined to their own AppDomain. However, the Visual Studio debugger needs a process to attach to, and asking it to debug itself clearly doesn’t make much sense! So we needed an external process for hosting template execution when debugging a template which, short of spinning up an entire second Visual Studio, brings with it some challenges in ensuring identical behavior in this new environment.

For simple templates - ones that aren't host-specific - there are unlikely to be any differences in behavior. Host-specific templates, though, particularly those using the Visual Studio automation object (or DTE) can find themselves very aware of these differences. Two specific issues we ran into were Visual Studio being busy with user interaction during calls made from our hosting process to the DTE and casts of COM objects not getting translated into the proper QueryInterface call after going through the .NET Remoting boundary across processes. Neither of these could occur on a template running inside Visual Studio, yet they would reproduce when attempting to debug the following template:

<#@ template hostspecific="true" language="C#" #>
<#@ assembly name="Microsoft.VisualStudio.Shell.Interop.8.0" #>
<#@ assembly name="EnvDTE" #>
<#@ import namespace="Microsoft.VisualStudio.Shell.Interop" #>
<#@ import namespace="Microsoft.VisualStudio.TextTemplating" #>
<#@ output extension=".txt" #>
<#
EnvDTE.DTE dte = ((IServiceProvider)Host).GetService(typeof(SDTE)) as EnvDTE.DTE;
EnvDTE.ProjectItem templateItem = dte.Solution.FindProjectItem(Host.TemplateFile);
foreach (EnvDTE.ProjectItem projectItem in templateItem.ContainingProject.ProjectItems)
{
#><#= projectItem.Name #><#
}
#>

This was an important scenario to us as many useful templates make heavy use of these features such as the templates packaged with Entity Framework and the T4Toolbox. We were able to solve the first by ensuring the template generation was initiated on its own STA thread. The second issue required us to intercept the call to obtain the DTE using a custom proxy in the host process and switch to COM remoting instead of marshaling via .NET remoting.

OK, now you’ve probably learned more than you ever thought you wanted to know about how T4 debugging works!  We hope you’ll find these improvements to T4 let you have a great experience debugging a wide range of templates with Visual Studio 2012 Update 1 and 2.

An oft-cited criticism of template-based code generators is that, over time, the templates tend towards spaghetti code; without some effort I think it’s a pretty valid stick to beat T4 and its ilk with.  But here’s the good news:  with not much effort, you can build up a library of chunks that you can compose to get radically more readable and quicker to produce templates.  The common analogy here is Lego (or Legos as my American colleagues endearingly refer to them), but actually, I’m going with bricks and mortar as my analogy.

OK, let’s look at an example here to see what I’m talking about.  Here’s a template I’m using at present to generate all the boilerplate code necessary to create new serializable Exception classes.

This generates a bunch of files with code like the example below (click through for the full code – it’s a bit large to insert)

This template is following the pattern of setting up a simple data structure to drive the code generator and then walking that structure.  In this case, it’s using the CodeModel from Visual Studio’s DTE automation interface to find exception classes marked with a specific attribute in the current project, and then populating a simple enumerable.   It then walks that enumerable, creating a new file for each exception, and populating it with a nice header, a namespace and the exception partial class.  Normally, all that logic would make for a complex template, where the exception code would be a bit hard to find when you actually wanted to change the generated output.  Here, with the inclusion of some pre-canned functionality and a procedural decomposition style, it’s pretty easy to see where the header, the namespace and its associated using statements and the main exception code all get written.  At this point, OO purists might be wearing a somewhat pained expression, but I’m really applying principals we normally associate with OO, such as single responsibility principle and applying them to my procedural world here (at least as far as the included helper files at the top of our example go.

What we end up with as input to this template is a very trivial handwritten exception declaration, and the code generator takes care of everything else.

So what’s the typical structure of such a helper? Let’s take a look at the simplest of the four I use here, the helper that lets me output a standard code gen header, StandardHeader.cs.t4:

As you can see – very simple, and very single purpose.  It’s just a definition in a class feature block of a method to write out a header.  A lot of my helpers are just like this – either one method or a small group of related methods to spit out common chunks of boilerplate, perhaps with a few simple parameters.

The key point here is that although the class feature block defines the method signature in C# control block code, it easily switches back to encapsulating boilerplate code without reverting to using clumsy “this.WriteLine()” methods.  This is one of T4’s strongest points, allowing me to very easily modify that header code with a simple cut and paste from a newer example that I like better.

OK, this is getting to be a long post so I’ll take a break.  Next time I’ll look at the Exceptions.cs.t4 helper and talk about keeping helpers decoupled using the approach I described as bricks and mortar.

Gareth Jones – Architect – Visual Studio

• T4 template debugging

In past releases of Visual Studio, debugging T4 templates has been a little more work than we’d like, so we’ve addressed that in this release with a simple to use but full-featured debugging experience:

You’ll see the ‘Debug T4 Template’ menu item above on the context menu for the template in the Solution Explorer.  There’s no need to set the debug flag on the template directive or any other special magic; just set a breakpoint in your T4 template and launch the debugger.  You’ll have full access to step through your template code and look at variables in the locals window.

There’s also new service interface, IDebugTextTemplating, to allow folks who use T4’s Visual Studio API to call T4 asynchronously under the debugger and so build equivalent debugging facilities into their tools, plus a couple of new support pieces under the hood to enable all this whilst maintaining backward compatibility on our existing interfaces.

I hope you enjoy the new productivity of debugging your T4 templates in our release candidate!

Everything below is in addition to the features and changes I noted in my previous post, What’s new with T4 in the Developer Preview of Visual Studio 11?

OK, here goes…

1. Directive processors are MUCH easier to build and more flexible
   Directive processors are T4’s plug-in mechanism.  They allow you to inject code into a template to pass and initialize parameters, or provide standard constructs within a template in a language-agnostic manner. However, they’ve been a bit fiddly to set up in the past and haven’t seen a lot of use.  This time, we’ve made them much cleaner and easier.
   • Directive processors can now be deployed as simple MEF components in a standard Visual Studio VSIX container. This means you can easily publish them on the Visual Studio gallery. There are new attributes [DirectiveProcessor] and [SupportedDirective] to make this easy. Look out for an example soon.
   • Directive processors can now be built to run across Visual Studio versions as they only need to implement the new immutable interface type, IDirectiveProcessor.
   • Directive processors can now add custom attributes to the underlying T4 class.  For example, it’s now really easy to create a directive processor to decorate a runtime template with MEF exports.  You simply implement the new method GetTemplateClassCustomAttributes().  I expect Kathleen to be all over this one!
2. Cleanliness features for runtime templates
   We’ve finished the name transition from Preprocessed to Runtime templates in the beta including renaming the item templates and, as we’re now seeing a lot more usage of runtime templates, we’ve added a few cleanup features.
   • You can now control the visibility of the generated class, using the new visibility property on the <#@ template #> directive with values public or internal.  This means that if you’ve got runtime templates as part of a published assembly, they no longer have to be part of your public API.
   • You can now turn off line pragmas for runtime templates with the new linePragmas property on the <#@ template #> directive with values true or false. This gives you more debugging options, as you can choose to either debug the original template or the generated code. This can also help if you’re finding the absolute filenames in pragmas are causing distracting merges under source code control.
3. UI Simplification
   As part of Visual Studio 11’s overall effort to simplify the UI, we’ve moved the Transform All Templates button out of the previously cluttered Solution Explorer toolbar.  You’ll now find it on the build toolbar and the build menu.  It also blocks the UI a bit less than it used to when you have lots of templates in your solution.

We hope you enjoy these new features.  As always, please give us feedback via Connect or UserVoice.

https://abstractiondev.wordpress.com/demo-videos/

He’s also not afraid to promote his ideas with conviction, which is refreshing these days!

In among all the excitement of new ALM features like integrated Code Review and MyWork to manage your daily workflow in the developer preview, there are, of course, a few nice new features for T4.

First of all, a naming change that's underway.  We've heard from you that 'Preprocessed' isn't the clearest choice of name for our templates, so we've switched to a new naming scheme:

• Design-time templates - Templates that run in the VS IDE to produce output based directly on the template content
• Run-time templates - Templates that are preprocessed in the VS IDE to produce a class that you incorporate into your application for use at runtime. Calling a method on the class produces output based on the template content.

For the Developer Preview, you'll see this change mostly in our documentation; the new terms will find their way into the product UI in time for Beta.

Now on to new features we've added in the preview:

1) Improved template inheritance for runtime templates that need access to features of the host

Previously, if you had a runtime template that declared itself as host-specific,

<#@ template hostSpecific="true" #>

it was hard to reuse the template through inheritance cleanly.  Derived templates that also declared host-specific would create a second 'Host" member that would hide the one from the base class, giving a warning.
To address this, we've added a third value for host-specific, "TrueFromBase", that makes the engine generate a host specific template, but expects the 'Host" property to come from the base class.

We've also made the comments generated from runtime templates more helpful.

2) Better error processing for design-time templates

When your design-time template gets past the compile stage and starts to run the compiled code, T4 now accurately reports line numbers for any exceptions that get thrown.
Don't forget to set debug="true" to make sure the compiler is emitting line number information.

3) COM Service helper

For folks using host-specific templates to access VS services, there've been a few reports of exceptions being thrown for one or two specific services.
Typically, it turns out that these services are written in managed code, but exposed over COM and then marshaled to T4's AppDomain over .Net Remoting. (Phew!)

This rather complex path leads to the exceptions, so we've included a simple extension method on IServiceProvider that you can use to get hold of those services and force them to behave properly:

    public static class ServiceProviderExtensions
    {
        public static Object GetCOMService(this IServiceProvider provider, Type type)
    }

Just call this method instead of the regular GetService() method and your COM troubles should be over.

4. Item templates now support LINQ out of the box.

We've updated our VS item templates, so when you add a new T4 template to your VS project, you'll automatically get all the directives you need to use LINQ right off the bat.

<#@ assembly name="System.Core" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.Collections.Generic" #>

5. Correct disposal of MEF container when T4 AppDomain recycles.

We provide a simple helper method (CompositionServices.InitializeServices() ) to set up a MEF composition container in the T4 AppDomain if you want to use VS' MEF components from your template.

T4 has a heuristic that decides when to recycle the secondary AppDomain for design-time templates. This one is really more of a bug fix, but I'll mention it just because not many folks know about the MEF setup helper.
The fix is that now when the AppDomain gets recycled, the MEF container is correctly Disposed.

We'll have some more features to talk about once the Beta comes along, but in the mean time, I hope you enjoy these improvements to T4 in our Developer Preview.

We’ve now got a third contender in the Visual Studio T4 Editor space.  The good folks over at DevArt have joined the race. From their press release:

Devart today releases a new powerful Visual Studio add-in for editing T4 templates with syntax highlighting, intellisense, code outlining, and all features of first-class text editor add-in for Visual Studio. It provides very high performance and makes creating T4 templates easier and faster.

With this new add-in, Devart offers a fast and easy way to create and edit T4 templates with multilevel template including, convenient template navigation, and rich code editing features.

So here’s my updated table of T4 editors (still in strictly alphabetical order)

Keep on transforming those templates.

A few weeks ago, I recorded a podcast about T4, and how it’s being used inside and outside Microsoft with the estimable David Starr, who I happened to meet up with at the recent MVP Summit.

You can check it out in the PluralSight Pluralcast blog. Enjoy.

https://www.pluralsight-training.net/community/blogs/pluralcast/archive/2011/04/07/pluralcast-39-t4-with-gareth-jones.aspx