Jeremy Kuhne's Blog

Custom directory enumeration in .NET Core 2.1

JeremyKuhne — Fri, 09 Mar 2018 01:59:18 GMT

As discussed in my previous post, this post will cover the new enumeration extensibility points. The extensibility API we're providing is meant to allow building high-performance custom enumerators. The performance gains primarily come through utilizing the new Span types and ref structs to allow relatively safe access to native data, avoiding unnecessary allocations. The API has been designed to be as simple as possible while still maintaining the low-allocation, high-performance primary design goals.

FileSystemEntry

File system data is wrapped in the System.IO.Enumeration.FileSystemEntry struct. It provides the same set of data that the FileSystemInfo classes do, along with some enumeration specific data:

FileSystemEnumerable

There are a few ways to get FileSystemEntry data. The simplest way is through the System.IO.Enumeration.FileSystemEnumerable class: Basic usage isn't too complicated. Let's suppose you wanted to just get the names (not the full paths) of all the files and directories in a given directory: You can, of course, write the equivalent functionality using the existing APIs that return FileSystemInfo objects (e.g. getting .Name off of each), but it comes with significant cost. Writing a custom enumerable only takes a single allocation for each MoveNext (the filename). Getting the Info objects via existing APIs will allocate much more (the Info class, the full path, etc.) and will be slower, particularly on Unix as it will cause another fstat call to fill out data you won't ever use. To take it another step, lets filter to just files (no directories): Here is another, potentially more useful, example. Lets say you want to create a helper that allows you to get all files with a set of given extensions: Let's suppose you want to count the number of files in a directory. With the new APIs you can write a solution that cuts allocations by 200x or more (yes, by a factor of 200). The example is a little strange as we need some sort of output transform. I picked int as the type and returned 1, but it could be anything, including string and string.Empty or null. Very similar to the above, we could total up file sizes. The performance characteristics of this last example differ quite a bit from Unix to Windows. Allocations are roughly equivalent, but Unix does not provide length when enumerating directories. We have to make another call to get the file length. .NET being designed for Windows, the content of what is in FileSystemInfo (and therefore matches what Windows gives back during enumeration). Unix doesn't give much more than the file name. To allow for this a number of the properties are lazy in the Unix implementation. Time stamps, length, and attributes are all in this bucket. Even the filename is lazy in that we don't convert the raw UTF-8 data to char until you access it. These details are important to know for multiple reasons:

Calling properties you don't strictly need can have non-trivial cost
Sitting on the struct without calling properties can give different results depending on how long you wait (notably with time stamps- we go out of our way to keep attributes constant)
Using IsDirectory and IsHidden is better than using Attributes to check those states

FileSystemEnumerator

FileSystemEnumerable is a simple IEnumerable wrapper around FileSystemEnumerator. FileSystemEnumerable is meant to be simpler and provide a model that limits the number of types via delegate usage. FileSystemEnumerator has a few more complicated options that can be used for even more advanced scenarios. Using the enumerator directly you can (if desired) more easily track when directories finish and have more control over errors. As the errors are native error codes, you would need to check the platform in addition to the code to know how to respond. It is not intended to be easy or commonly needed.

FileSystemName

This helper class provides filename matching methods. Both Matches* methods allow escaping with the forward slash character '/'. MatchesWin32Expression() is a little complicated. It matches according to [MSA-FS] 2.1.4.4 Algorithm for Determining if a FileName Is in an Expression. This is the algorithm that Windows actually uses under the covers (see RtlIsNameInExpression). If you want to match the way Win32 does you first have to call TranslateWin32Expression() to get your '*' and '?' translated to the appropriate '>', '<', and '"' characters. It is there if you want to match in the Win32 style. MatchesSimpleExpression() is the recommended style of matching. Win32 rules aren't easy to intuit. That's the summary of the changes we've introduced in 2.1.

FAQ

Why is this so complicated?

This isn't intended to be a common-use API. The existing APIs will be kept, maintained, and extended based on customer demand. We don't want to:

Have scenarios blocked waiting for new APIs to work their way through the system
Have to write "normal" APIs to address more corner cases

In order to make this a usable extension point we have to sacrifice some usability to get the necessary characteristics. Note that what people build on this will directly impact our future designs of the standard, usability focused, APIs.

Why are you using Linq in your examples?

For example clarity. Some of the examples above could be optimized further.

Why aren't you providing an `X` matcher?

We want to only provide matchers that have broad applicability. Based on feedback we can and will consider adding new matchers in the future.

How do I get other platform specific data?

This is something we're investigating for future improvements. We might, for example, expose the UTF-8 data via another interface off of the entry data (or some other mechanism).

System.IO in .NET Core 2.1 sneak peek

JeremyKuhne — Thu, 08 Mar 2018 01:46:22 GMT

Now that .NET Core 2.1 Preview 1 has released it is time for a preview of what you can expect to see in Preview 2 and beyond in the System.IO namespace. The 2.1 release changes are the largest changes in quite some time.

Goals

Cross platform Code should run consistently as possible between platforms. You should also be able to successfully write code that works in mixed environments. Accessing a Unix volume from Windows should just work, for example. Light touch System.IO's aggressive handholding blocked a number of scenarios, including mixed environments as described above. Historically this was driven partially due to Code Access Security (CAS) in NetFX (e.g. 4.x). It was also a more plausible approach when .NET was a Windows only solution to aggressively attempt to predict OS behavior. .NET judging the validity of paths is the key example of this well-intentioned, but heavy hand. We now strive to not replicate or predict the outcome of OS API calls. Performant This is always important to us. We want to put as little overhead on your platform and enable building performant solutions. Flexible We don't want to have an API for every single conceivable scenario, but neither do we want to make it impossible/difficult to build solutions. Part of addressing this is just being flexible and forward-thinking in the API set in general. Another part of this can be addressed by providing extension points that allow you to build complicated solutions in a performant way without resorting to P/Invoking directly into the underlying platform.

Overview

What are you getting from this release? Here are the highlights:

More consistent behavior cross platform
Fixed edge cases, particularly on Windows
New Span APIs in System.IO.Path
Path.Join() APIs that don't have root checking behavior like Path.Combine()
Path.GetFullPath() overload that avoids using the current working directory
A number of new directory enumeration options
1. Filtering out specific attributes
2. Simple matching behavior
3. Ignoring inaccessible files
4. Specifying buffer size (notably for Windows UNC scenarios)
A low-level enumeration extensibility API
Significantly faster directory enumeration
1. Typically 2-4x faster on Windows (Unix 1.3x - 1.4x)
2. Significantly lower allocation counts (2x - 40x lower, GC collections cut 8x+)
Faster Path APIs (GetFullPath() is now 2x as fast as .NET Core 2.0)

Key behavior changes

We've changed behavior to fix issues. Here are the key impacts:

Exceptions for "bad" paths are thrown when used, not when normalizing (as we can't know what is "valid" without using them)
Cross plat scenarios are no longer blocked (mounting Unix volumes/shares on Windows for one example)
Performance is significantly increased
Behavior is more consistent when running code across platforms

In more detail this means:

We've made matching consistent with legacy behavior on non-Windows platforms (????.txt matches the same on all platforms)
Match expressions no longer match 8.3 file names on Windows (*.htm will match only *.htm, not *.html, *.htmz if the volume 8.3 generation on)
We've fixed enumeration of unusual Windows files - you can now get *Info classes out successfully and use the methods on files that end in spaces, periods
We do very little validation of paths up front as there is very little we can accurately predict (unblocking numerous xplat and some existing Windows scenarios)
We only check for embedded nulls, no other chars are rejected, including wildcards (as nulls are never supported and OS APIs almost universally take null terminated strings)
- We don't check for "proper" colon placement
- We don't check for segment length or total path length
- We don't check for "proper" UNCs
- We still throw for null or empty paths on all platforms or paths of all spaces on Windows (as they throw from the Win32 GetFullPathName)
We don't trim leading spaces on any paths on Windows anymore (we did for some, not others)
We don't trim whitespace characters from the end of paths on Windows (such as nbsp)
GetPathRoot, GetDirectoryName, etc. don't throw for empty anymore, they return null (like they do for null).
GetPathRoot now works consistently with various Windows prefixes (e.g. \\?\, \\.\)

New System.IO.Path APIs

The majority of what we have here are new ReadOnlySpan overloads, which allow you to avoid unnecessary string allocations. Spans are a key new feature in .NET 2.1. You can pass strings as spans using the .AsSpan() extension (there is an implicit conversion as well). ReadOnlySpan has a number of extensions that allow you to evaluate it as you would a string. Join() APIs allow putting path segments together without analysis of segment rooting. Combine("C:\Foo\", "\Bar") gives you "\Bar". Join gives you "C:\Foo\Bar". Path.GetFullPath(path, basePath) is an important addition. Normalizing paths that aren't fully qualified (e.g. relative to the current directory) is dangerous and discouraged. The reason is that the current directory is a process-wide setting. Getting into a state where a separate thread unexpectedly changes the working directory is common enough that you should make every attempt to not use the existing GetFullPath().

New EnumerationOptions

There have been numerous asks for directory enumeration options over the years. To fulfill some of those requests and allow addressing more we're introducing the EnumerationOptions class. The existing Directory and DirectoryInfo enumeration methods now have overloads that take this new class. This class has some defaults that are different than historical behavior:

Matching is simple- '?' is always one character, '*' is always 0+, "*.*" is any filename with a period
Hidden and system files are skipped by default- typically one doesn't want the .vs, .git, $RECYCLE.BIN, Thumbs.db etc. folders in results.

You can, of course, choose any setting you want.

New enumeration extensibility model

Those options not enough? Now you can write your own fast, low level enumerators. Want your own matching algorithm? Want to total file sizes? Want to match all files with a set of extensions? Almost anything is possible with the new API that live in System.IO.Enumeration. It is a large topic, so I'll address it in the next post.

FAQ

What exceptions do I get from bad paths now? Whatever the OS tells us, we'll tell you. This will always be some sort of IOException for a bad path. Sometimes it might be a File/DirectoryNotFound. The key thing to remember is that you get exceptions when you try to use the path. GetFullPath() doesn't throw for these as it has no practical way of checking. What if I still want to check invalid characters? You can do this manually using GetInvalidPathChars(). It isn't recommended as it isn't always correct on any platform. You may have NTFS/FAT volumes mounted in Unix or vice-versa. Where did the speed improvements come from? Primarily from reducing allocations. Keeping work on the stack and out of the heap can have a dramatic impact on large sets. Some wins come from smarter use of available OS APIs. A chunk comes from not validating paths on every single API call. When does AttributesToSkip get applied? First thing. One effect of this is that filtering out FileAttributes.Directory makes the RecurseSubdirectories option meaningless. What about other file matching types? We definitely want to add more in the future. Things like globstar (**), POSIX pattern matching notation (glob), possibly regex. With the extensibility APIs writing a custom matcher is easy. If you're passionate about any of these we're always open to contributions. Why did you change enumeration defaults? Note that we only changed them when you use the new EnumerationOptions class. Existing APIs should behave as they did (modulo the matching consistency fixes mentioned in the post). We picked new defaults based on OS defaults (shell & command line) and what one would expect when enumerating end-user files (e.g. to present to users). These obviously won't be right for all scenarios, but changing the settings is easy. What about the more obscure Windows matching characters? '<', '>', and '"' were unblocked in 2.0. They're still supported, but only on Windows through the normal APIs. You can, if you want, use them on Unix if you use the extensibility points described in the next post.

Wildcards in Windows

JeremyKuhne — Sun, 04 Jun 2017 21:15:34 GMT

I put forth the challenge in my last post about how to match files that start, end, or simply have a period in them. It isn't easy to do. Wildcards in paths seem simple at first. You have '?' to match any single character and '*' to match zero or more of any characters. Matching in reality doesn't exactly match up. '*' gives you all files. '?' gives you all files that are just one character. '*a*' gives you all files that have an 'a' in them. '*_*' gives you all files that have an underscore in them. '*.*' gives you ... all files. The seemingly odd behavior of matching dates back to DOS days. Every file was in 8.3 form: 8 characters for the name, a period, and a 3 character file extension. You couldn't use a period in the filename or extension- it was always the separator for the two filename components. (The period wasn't even stored on disk.)

MS-DOS was based on QDOS (later 86-DOS), which was written by Tim Patterson for Seattle Computer Products for their 8086 computer kit. It was designed to be compatible with the dominant microcomputer OS of the late 70s, CP/M. Both QDOS and CP/M had 8.3 filenames and used similar wildcard matching with ? and *. CP/M was inspired primarily by DEC OSes, which had the same style filenames and wildcard matching. (Most DEC OSes had 6.3 filenames, and also used the colon ':' to specify devices, akin to 'C:'.)

With the advent of 32 bit Windows (Windows 95 and NT), 8.3 filenames were no longer a limitation. You could now have filenames that were up to 255 characters, and the period was now a legitimate character. After 25 years or so of '*.*', keeping similar semantics made a lot of sense. Here are some of the common patterns in DOS:

Pattern	Meaning
`.`	All file names, all extensions (all files).
`*.`	All file names without extensions.
`foo.`	All files beginning with "foo".
`foo??.*`	All files beginning with "foo" and up to two additional characters.

The real rules were subtle and had some "surprising" behaviors that were not carried forward to Win32. Raymond Chen describes the algorithm in "How did wildcards work in MS-DOS?". So how exactly do wildcards work now? The patterns mentioned above work as expected, but how? It obviously isn't exactly correct that '?' matched any single character and '*' to match zero or more of any characters... The answer lies in that there are actually five wildcards in Windows:

Wildcard	Meaning
`*`	Zero or more characters.
`?`	Exactly one character.
`<`	Matches zero or more characters until encountering and matching the final period in the name. (DOS_STAR)
`>`	Matches any single character, or zero if to the left of a period or the end of the string- or contiguous to other `>` that are in said position. (DOS_QM)
`"`	Matches a period or zero characters at the end of the string. (DOS_DOT)

If these are the rules, why don't they work that way? The answer is that Win32 implicitly changes your wildcards. This is the another part to the mystery of wildcards. The rules are as follows:

All '?' are changed to '>'.
All '.' followed by '?' or '*' are changed to '"'.
A path ending in '*' that had a final period before normalizing is changed to '<'.

The last rule brings up the last piece of the wildcard puzzle. As described in my previous posts, path normalization eats trailing periods and spaces. This can make it difficult to search for filenames with trailing spaces/periods. Now that we have the details of wildcard matching the challenges in my last post can be answered.

List files ending with a space

You need to skip normalization for this to work. dir "\\?\C:\* " will give you all files that end with a space in "C:\".

List files starting with a period

This is actually pretty simple. dir .* will give you all files that start with a period in the current directory.

List files with a period

This involves using the "DOS" wildcards directly (which need to be escaped at the command line). dir *.^< will give you all files that have a period in the current directory. "*.<" is necessary as, per the rules, "*.*" would become "*"*" which is equivalent to "*".

List files ending with a period

You need to skip normalization for this to work. dir \\?\C:\*. will give you all files that end with a period in "C:\".

APIs in play

Searching for files always goes through one of the FindFirstFile APIs. FindFirstFile will normalize the path, cutting it into the directory portion of the path and the filename. The filename wildcards are munged as explained above. FindFirstFile opens a file handle on the directory and calls NtQueryDirectoryFile with the altered filename. (Full details of the API are in the kernel mode version ZwQueryDirectoryFile.) When the filesystem driver responds to the query it will use FsRtlIsNameInExpression to do it's matching. RtlIsNameInExpression is the user mode version of this API. (Note that the documentation can be a little misleading. When you do a case-insensitive search and don't provide a translation table, the name is converted to uppercase. The expression (pattern) is not- you have to do this manually. RtlUpcaseUnicodeString is the Win32 API for this.) If you really want to use "*" and "?" you could potentially call NtQueryDirectoryFile directly. You won't get your existing search string changed, but all five wildcards are processed, of course.

Wildcard challenge

JeremyKuhne — Wed, 31 May 2017 05:38:15 GMT

Finding files with spaces from a command prompt might not be immediately obvious. Using quotes is the answer.

List all files with a space: dir "* *"
List all files that start with a space: dir " *"
List all files that end with a space: ???

#3 is a bit more challenging. Normalization has something to do with the answer. The real challenge is how do you find files with periods??

List all files with a period: ???
List all files that start with a period: ???
List all files that end with a period: ???

Can they all even be done? If not, why? If you have an answer, why does it work?

Creator's Update, .NET 4.7 and long paths

JeremyKuhne — Wed, 31 May 2017 00:56:51 GMT

Most notable bit is CMD supports long paths now. You still need to enable the long path policy as it isn't on by default. .NET 4.7 had some minor tweaks. It now allows using device syntax (\\?\) with FileIOPermissions as long as you are running in full trust. This should allow you to use \\?\ paths everywhere (classes that asserted permissions outside of mscorlib didn't work in 4.6.2). I also fixed a bug on my part in 4.7. (doh)

.NET 4.6.2 and long paths on Windows 10

JeremyKuhne — Sat, 30 Jul 2016 01:21:22 GMT

The Windows 10 Anniversary update is almost out the door. .NET 4.6.2 is in the update (as we've looked at in the past few posts). I've talked a bit about what we've done in 4.6.2 around paths, and how that is targeted at both allowing access to previously inaccessible paths and opens up the door for long paths when the OS has support. Well, as people have discovered, Windows 10 now has started to open up support. In this post I'll talk about how to enable that support.

Enabling Win32 Long Path Support

Long paths aren't enabled by default yet. You need to set a policy to enable the support. To do this you want to "Edit group policy" in the Start search bar or run "gpedit.msc" from the Run command (Windows-R). In the Local Group Policy Editor navigate to "Local Computer Policy: Computer Configuration: Administrative Templates: All Settings". In this location you can find "Enable Win32 long paths". [caption id="attachment_525" align="alignnone" width="754"] Enabling Win32 long paths in the policy editor.[/caption] After you've turned this on you can fire up a new instance of PowerShell and free yourself from the constraints of MAX_PATH! The key File and Directory Management APIs respect this and now allow you to skip the check for MAX_PATH without having to resort to using "\\?\" (look back to my earlier posts on path formats to understand how this works). This is also possible as PowerShell has opted into the new .NET path support (being that it is a .NET application). If you look carefully at the description in the setting you'll see "Enabling Win32 long paths will allow manifested win32 applications...". That's the second gate to getting support- your app must have a specific manifest setting. You can see what this is by opening C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe in Visual Studio or some other manifest viewer. Doing so you'll see the following section in it's manifest:


  
    true

These two gates will get you the native (Win32) support for long paths. In a managed app you'll also need the new behavior in .NET. The next section covers this.

Configuring a Simple Long Path .NET Console App

This example uses a new C# Console Application in Visual Studio 2015. The first thing to do after creating a new console app is edit the App.Config file and add the following after the end tag:

If you have the 4.6.2 Targeting Pack installed (from the 4.6.2 Developer Pack) you can alternatively select 4.6.2 as your target framework in the project properties instead of using the app.config setting. The defaults for these two values are true if the target framework is 4.6.1 or earlier. The second thing to do is add the Application Manifest File item to your project. After doing so add the windowsSettings block I shared above. In the default template there is already a commented-out section for windowsSettings, you can uncomment this and add this specific longPathAware setting. Here is a sample block to add to your Main() method to test it out:

string reallyLongDirectory = @"C:\Test\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
reallyLongDirectory = reallyLongDirectory + @"\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
reallyLongDirectory = reallyLongDirectory + @"\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";

Console.WriteLine($"Creating a directory that is {reallyLongDirectory.Length} characters long");
Directory.CreateDirectory(reallyLongDirectory);

You can open up PowerShell and go and look at the directory you created! Yayyyyyy! This is the start of what has been a very long journey to remove MAX_PATH constraints. There is still much to do, but now the door is finally open. The rest can and will come- keep your feedback coming in to keep us on track! The Windows support is documented on MSDN. APIs that aren't listed may work- if the API took \\?\ in the past it will likely support the new behavior. Note that in this initial release CMD doesn't support long paths. The Shell doesn't add support either, but previously had limited support utilizing 8.3 filename trickery. I'll leave it to the Windows team for any further details.

More on new .NET path handling

JeremyKuhne — Tue, 21 Jun 2016 23:39:20 GMT

In my prior post I talked briefly about the new path handling in .NET 4.6.2. In the latest drop (14367) PowerShell opts into both of the new behaviors so you can easily play around with the changes. There are two basic changes that can be controlled via two compatibility switches. The changes both originated in CoreFX so you can get a more detailed look if you so choose. I'm going to talk about these this week on On .NET, so bring your questions.

Normalization Changes

The first change is that we now try to defer to the OS to normalize paths. Trying to emulate what the OS does led to some discrepancies- one notable one being that you couldn't do anything with directories that end in a space. You can create directories with a space at the end from the CMD prompt, but they were impossible to do anything with in .NET. Refer back to my post on normalization to see a bit more about what is/isn't possible in Windows. The second change is that we allow better access to DOS device paths. This includes both the normalized (\\.\) and non-normalized (\\?\) syntaxes. These were poorly documented and subsequently poorly understood. Handling was spotty and inconsistent. Getting at paths that were possible using non-normalized syntax (such as \\?\C:\foo.) used to be impossible in .NET. Both of the DOS device path syntaxes skip most character validity checks. Why? Those checks were wrong and essentially impossible to get correct. Named pipes, for example, can use many characters you can't use in a "normal" NTFS path (and do, Nvidia creates a pipe like this on my machine). This sort of validation is better left to the OS and the various drivers involved, so we do. The extended DOS device syntax will only work for core APIs that live in mscorlib if you're in full trust. FileIOPermissions historically is the class responsible for kicking \\?\ paths and still is. Comparing extended paths is too costly to do in FileIOPermissions (partially because paths don't have to exist, which makes the logic crazy). For this release the internal usages in mscorlib have a special out when we're running full trust. This should cover the vast majority of usages. I'm looking into expanding this to all usages of FileIOPermissions in full trust in a future .NET. (Note that there are no such limitations for CoreFX as there is no limited trust model.) Normalization changes are controlled via the UseLegacyPathHandling switch, which is off by default if you target 4.6.2 or higher (the semantics here are part of the AppContext design- you opt into backwardly compatible behavior on newer Framework versions).

Long Path Changes

The key driver in re-evaluating the normalization was the need to support long paths (paths over MAX_PATH, or 260 characters). The old logic didn't scale and prevented usage of the current "long path" solution on Windows, extended DOS device paths (\\?\). In 4.6.2 we will no longer throw PathTooLongException if we see a path that is >= MAX_PATH. If the OS doesn't like it we'll surface whatever the OS returns as an error (which may be PathTooLong), but we won't second guess what the OS will do. This enables you to use extended long paths and, if the OS would let you, regular long paths. *cough* :) (Note that in many cases the OS will return DirectoryNotFound for paths that are too long. The primary case is with creating files. We don't have a way to programmatically check whether paths are actually too long so we can't translate DirectoryNotFound into PathTooLong. We're looking at updating the exception text in the future to say it "may also be caused by a path that is too long" where such a thing is possible.) In CoreFX, .NET will put the extended syntax on for you when calling Windows APIs. This allows you to use long paths without any extra thought or new OS support (if it should ever arrive *cough* *cough*). This implicit handling isn't done in 4.6.2 as it was to costly to fit in the schedule (it is a much more difficult task to get this implicit support in the desktop code as opposed to CoreFX). The long path changes are controlled via the BlockLongPaths switch, which again is off by default if you target 4.6.2 or higher.

Config File Setting

If you target 4.6.2 this isn't necessary, but if you want to enable the behavior for existing code, here is config file snippet you need to use.

New .NET path handling sneak peek

JeremyKuhne — Thu, 09 Jun 2016 05:25:08 GMT

As mentioned I'm going to do the On .NET show in a couple weeks to talk about path changes in .NET. In the meantime, a little bit of fun for you if you've got the latest (14361) fast ring build for Windows 10 Anniversary Update... UPDATE: In the latest drop (14367) PowerShell opts into the new behaviors- the following config file isn't necessary. Drop a file called powershell.exe.config in C:\Windows\System32\WindowsPowerShell\v1.0 with the following contents to opt in to one of the new 4.6.2 features (as PS doesn't target 4.6.2):

Now start up PowerShell and create yourself a C:\foo directory so you can try a few things. Make yourself a foo file: PS C:\foo> echo footext > foo And a bar file: PS C:\foo> echo bartext > bar Copy foo to bar:foo: PS C:\foo> copy foo \\?\C:\foo\bar:foo Take a look at what we've got: PS C:\foo> dir \\?\C:\foo Now take a closer look: PS C:\foo> type bar PS C:\foo> type -LiteralPath \\?\C:\foo\bar:foo I'll leave it to you to play around a bit more with this. Look at my prior posts on path handling to clarify what is going on here and try and give me some of your own cool examples of what you can do. :)

Using MSBuild properties in T4 templates

JeremyKuhne — Tue, 07 Jun 2016 01:09:21 GMT

The Problem

Text Templates in VS are cool, but they have some rough edges. One in particular is getting at project properties from your templates. One of the biggest problems is that the generator for the project build and the generator that runs inside VS are different. The project build generator will allow you to define your own parameters to send to the build task, but the VS generator won't read them. The VS generator will let you look at only 4 things:

"namespaceHint"
"projects" (returns a vertical bar separated list of projects)
"projectDefaultNamespace"
"projectName:OutputPath" (where projectName is the name of the project)

While you could match these names and values in your MSBuild project files it's pretty limiting. It's most unfortunate that the VS generator doesn't pick up .

A Solution

Utilizing a little-known ability to run arbitrary MSBuild tasks as a custom tool you can unify the behavior on the generator the project build uses. You'll need to make a small tweak to your project file to set this up.

Edit your .cs/.vbproj and include the CustomT4.targets at the end of the file.
Reload your project.
Change the "Custom Tool" property from "TextTemplatingFileGenerator" to "MSBuild:TransformAll".
Change hostSpecific="false" to "true" in the <#@template> directive.
Use this.Host.ResolveParameterValue(".", ".", "TargetPath") to get at the TargetPath property.

The simple test template looks like the following:

<#@ template debug="false" hostspecific="true" language="C#" #>
<#@ output extension=".txt" #>
Test "<#= this.Host.ResolveParameterValue(".", ".", "TargetPath") #>"

And here is the target file with a number of properties you're likely to want.

Notes

The first two arguments to ResolveParameterValue() are meaningless, but can not be empty strings despite the documentation. I decided to use period because it's unobtrusive.

VS Background Builds

JeremyKuhne — Mon, 06 Jun 2016 05:18:02 GMT

Two MSBuild related things that I keep losing track of every few years that are incredibly hard to discover.

Logging Background Builds

Visual Studio does a number of background (or "Design Time") builds on your behalf. They don't log anywhere by default. If you want to look at what is going on you need to "set TRACEDESIGNTIME=true" in your environment and launch VS. This will dump diagnostic builds to your "%temp%" folder in the form of "ProjectName_TargetName_unique.designtime.log".

Using a Build Task as a Custom Tool

Finding the documentation for Custom Tool is hard enough, let alone the fact you can invoke a specified MSBuild task. Set the Custom Tool property to "MSBuild:targetName" and a background build will fire whenever the file changes. You'll need to turn on the background build logging as described above to see what is going on. One last related note: IVSSingleFileCodeGenerator is the normal interface involved for Custom Tools.