https://github.com/KirillOsenkov/MSBuildStructuredLog

It is an MSBuild logger that can be passed to MSBuild during a build and it records information about everything that happens in the build. But instead of just dumping everything into a huge text file it preserves the structure and relationships between elements, and lets you save and open the structured log in .xml format. This has significant advantages to help you understand and navigate large build logs.

Give it a try and let me know what you think!

https://blogs.msdn.microsoft.com/jgoldb/2008/09/26/updated-wpfperf-performance-profiling-tools-for-wpf/

I had to ask around, but here are the links to download the tools (Windows Performance Toolkit Setup):

https://download.microsoft.com/download/A/6/A/A6AC035D-DA3F-4F0C-ADA4-37C8E5D34E3D/setup/WinSDKPerformanceToolKit_amd64/wpt_x64.msi

After installing this one, also download and run the patch:

https://download.microsoft.com/download/1/8/9/189A7832-49D8-4978-85E8-3DFFF44E6C04/WpfPerf_timezone_patch.msp

When the patch installer prompts, choose “Repair”.

Then you can launch WPF Performance Suite (which has two tools: Perforator and Visual Profiler):

The new release is here:

https://github.com/KirillOsenkov/ContentSync/releases/tag/v1.1

Now you can customize what operations to perform:

-c       Copy files from source that don't exist in destination (left-only).

-u       Update files that have changed between source and destination. This only overwrites the destination file if the contents are different.

-d       Delete right-only files (that are in destination but not in source).

-ds      Delete same files (that exist in source and destination and are same).

-dc      Delete changed files from destination (can't be used with -u).

-whatif  Print what would have been done (without changing anything).

-h         Help (duh!)

Examples:

Update only the files that exist both in source and destination:

ContentSync.exe C:\Source D:\Destination –u

This scenario is useful when you’ve selected a subset from a folder and copied it to a separate folder, but now realized that you want to copy the same subset to the new folder again from updated source.

This command will copy new files and update changed ones, but won’t delete any extra remaining files from destination:

ContentSync.exe C:\Source D:\Destination –cu

Additionally you can use –whatif for speculative mode, when it prints what it would have done without actually changing anything on disk:

ContentSync.exe C:\Source D:\Destination –d –whatif

This prints out all the files that are in destination but not in source (without actually deleting them from the destination). Just passing –whatif results in diffing the contents of both folders (it doesn’t actually show detailed file diffs, just shows which files are identical and which are different).

I wanted to just take robocopy options, but they didn’t map exactly to the concepts I wanted to represent. So I opted for a new argument syntax.

Finally it prints out a summary at the end:

https://blogs.msdn.com/b/dr._ex/archive/2007/04/17/using-enablevsiplogging-to-identify-menus-and-commands-with-vs-2005-sp1.aspx

Run this .reg file:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\14.0\General]
"EnableVSIPLogging"=dword:00000001

Then in Visual Studio hold Ctrl+Shift and click any menu, toolbar command or context menu item. You will get a message box with the info about the command (like below):

---------------------------
VSDebug Message
---------------------------
Command data:

    Guid = {1492DB0A-85A2-4E43-BF0D-CE55B89A8CC6}
    GuidID = 849
    CmdID = 1
    Type = 0x00000001
    Flags = 0x80000000
    Canonical name = View.C#Interactive
    Localized name = (Not set)
---------------------------
OK  
---------------------------

This is the safe and expected behavior, however there are whole classes of tools that track file modification based off the timestamp alone. For instance, MSBuild will trigger a cascading rebuild of all dependencies of a file if its timestamp has changed (even if the actual file contents is exactly the same). This is called overbuilding. A scalable build system should detect that the file contents didn’t change and avoid doing any work in this case.

Or take another example, suppose you need to upload hundreds of thousands of files to Azure using MSDeploy. If the timestamp on those files has changed, MSDeploy will upload those files even if the actual content is the same.

In general, if you’re deciding whether a file was modified based off the timestamp, you’re bound to schedule unnecessary work that could have been avoided if you checked whether the actual file bytes have changed.

Long story short, I wrote an open-source tool to sync/mirror directories based on the file content, not the timestamp:

https://github.com/KirillOsenkov/ContentSync

Usage: ContentSync.exe <Source> <Destination>

I needed this for the exact Azure website deployment scenario: I maintain https://source.roslyn.io, and we’ve recently moved from on-premises Microsoft hosting to Azure. I use https://github.com/KirillOsenkov/SourceBrowser to regenerate the website every night, and that’s 160,000 modified files totaling over 1 GB. You don’t want to upload that to Azure every night.

So I inserted an intermediate step into my deployment script – I ContentSync from the freshly built Index folder (it’s not incremental, all the files are generated from scratch every time) into a Staging folder from the last time. ContentSync doesn’t touch the files that haven’t actually changed (and that’s 99.9…%), and so MSDeploy doesn’t upload them since the timestamp is the same.

While I’m here, kudos to David Ebbo and his excellent https://github.com/davidebbo/WAWSDeploy that I use to simply deploy a folder to Azure (read more here: https://blog.davidebbo.com/2014/03/WAWSDeploy.html). It uses MSDeploy internally and hides all the details I don’t want to know about behind a super simple UX.

I’m now posting this from the brand new OpenLiveWriter which I just compiled and built myself!

https://github.com/OpenLiveWriter/OpenLiveWriter

Awesome!!!

So I took a part of Roslyn VB parser that parses XML literals, ported it to C# and made it into a standalone library that I could use for my purposes. Realizing that it might fill some gaps for certain scenarios out there I’m putting it out as open-source:

https://github.com/KirillOsenkov/XmlParser

And here’s an example of what I needed it for: notice the colored XML with hyperlinks?

https://referencesource.microsoft.com/#MSBuildFiles/C/ProgramFiles(x86)/MSBuild/14.0/bin_/amd64/Microsoft.CSharp.Core.targets,67

The parser is also available on NuGet:

Microsoft.Language.Xml

Microsoft.Language.Xml.Editor

The editor one is a simple language service for Visual Studio, which is less useful since Visual Studio already has a far more powerful language services for XML than this one. The advantage here however is that you can tweak this one any way you want to add further extensibility features to XML files in Visual Studio on top of what’s already there.

MSBuild could have had fine-grained parallelism where it could execute multiple tasks on different threads in parallel within the same process, significantly speeding up builds. However it can’t do that because many tasks rely on Environment.CurrentDirectory which is per process, and so to make sure the current directory is deterministic and well-known at the time a task executes, there can only be one task running at a time within the same process.

Additionally, it is terrible for discoverability and ability to reason about assumptions. Relying on Environment.CurrentDirectory makes an implicit assumption that is not recorded anywhere. For instance, here’s a field in Team Build that lets you select unit-test assemblies to run:

Where is this pattern rooted? To be able to write a pattern you need to know if its relative (to what?) and in which directory it is rooted. It obviously can’t be a full path since on the CI server the path varies by build agent id. After reading the source code for the build activity that calculates the files from this spec I found this:

This code relies on the current directory being set… where? To what? It’s a nightmare scenario for maintainers.

Last but not least, Environment.CurrentDirectory is global mutable state. Worse, it is global mutable state that makes it impossible to track causality. Who set this last? Where to put a breakpoint to intercept mutation?

I’m sure there are multiple reasons I’m missing but I don’t even care to enumerate exhaustively. The above is terrifying enough to send a clear message: avoid at all costs!

But, a reasonable response to that would be: what to do instead? Use patterns like $(SolutionRoot)\bin\Release\*.Tests.dll instead of just bin\Release. Every path should be rooted (maybe with a variable that is well defined) so that its clear where it is. Pass known variables around, avoid the untraceable global mutable state from the 80’s.

Same goes for environment variables. Avoid at all costs! If a build requires an environment variable set to work correctly, it has failed.

P.S. Also riddle me this. Why on earth is there a Directory.GetCurrentDirectory()?

But let us step back.

Let’s create a new C# Console Application in Visual Studio and build it with diagnostic verbosity. The most commonly useful tool to investigate build issues is the MSBuild diagnostic log:

msbuild.exe ConsoleApplication1.csproj /fl1 /noconlog /v:diag

(fl1 = write log to file msbuild1.log, noconlog = don’t write to console, v:diag = diagnostic verbosity)

When you open msbuild1.log and search for “csc”, you will see that the compiler invocation indeed includes a strange file:

csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /platform:anycpu32bitpreferred /errorreport:prompt /warn:4 /define:DEBUG;TRACE /highentropyva+ /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\Microsoft.CSharp.dll"  ... /debug+ /debug:full /filealign:512 /optimize- /out:obj\Debug\CS9.exe /subsystemversion:6.00 /target:exe /utf8output Program.cs Properties\AssemblyInfo.cs "C:\Users\kirill\AppData\Local\Temp\.NETFramework,Version=v4.5.AssemblyAttributes.cs"

Turns out, every time you build a C# project, MSBuild makes sure there’s a generated file in the TEMP directory called AssemblyAttributes.cs with content similar to this:

// <autogenerated />
using System;
using System.Reflection;
[assembly: global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETFramework,Version=v4.5", FrameworkDisplayName = ".NET Framework 4.5")]

But how does it get there?

Let’s open the excellent free MSBuild Explorer tool (https://msbuildexplorer.com) and open our .csproj file:

In the list of properties we find the TargetFrameworkMonikerAssemblyAttributesPath MSBuild property which is the same path that’s passed to the C# compiler. But who sets this MSBuild property and where?

Let’s now open https://source.roslyn.io which happens to include semantically hyperlinked MSBuild targets for C# and search for our property name: https://source.roslyn.io/#q=TargetFrameworkMonikerAssemblyAttributesPath.

We indeed find an MSBuild property with this name and can view its declarations and usages:

https://source.roslyn.io/#MSBuildProperty=TargetFrameworkMonikerAssemblyAttributesPath

We see the one place in the common targets where this property is written to:

https://source.roslyn.io/#MSBuildFiles/C/ProgramFiles(x86)/MSBuild/14.0/bin_/amd64/Microsoft.Common.CurrentVersion.targets,2928

Oh hey, look:

<TargetFrameworkMonikerAssemblyAttributesPath Condition="'$(TargetFrameworkMonikerAssemblyAttributesPath)' == ''">$([System.IO.Path]::Combine('$([System.IO.Path]::GetTempPath())','$(TargetFrameworkMoniker).AssemblyAttributes$(DefaultLanguageSourceExtension)'))</TargetFrameworkMonikerAssemblyAttributesPath>

This is the unfortunate place in MSBuild default targets that generates the file to the TEMP directory. Why is this bad? Well, it turns out, the location of TEMP may change if, for example, you use Remote Desktop to log on to your machine – the TEMP folder is different than your console sessions: https://blogs.msdn.com/b/oldnewthing/archive/2011/01/25/10119675.aspx

And so MSBuild sees that the file timestamp is newer than it remembers and decides to rebuild the project.

Let’s do an experiment. If you haven’t yet, go and set this registry key that I blogged about earlier. This will enable VS to tell you every time it decides to rebuild a project why it thought the project wasn’t up-to-date. Now build your console app once in VS (or in command line). Build again. Notice how it immediately says Build succeeded? Now go and edit that temporary file. Rebuild again – VS will decide that the project is out-of-date and rebuild it:

This is the problem that I mentioned in the beginning of this post. This breaks incrementality if multiple projects touch this file (and they often do!).

Fortunately for us, MSBuild is flexible enough so we can work around it. The good design is to generate this file into the Intermediate directory (usually called obj), because this is where all transient and temporary files should go during a build process. We can either set this property in our project file:

  <PropertyGroup>
    <TargetFrameworkMonikerAssemblyAttributesPath>$([System.IO.Path]::Combine('$(IntermediateOutputPath)','$(TargetFrameworkMoniker).AssemblyAttributes$(DefaultLanguageSourceExtension)'))</TargetFrameworkMonikerAssemblyAttributesPath>
  </PropertyGroup>

Or if your build uses a common .props file, set this property there. This will ensure that your build doesn’t depend on the TEMP directory and is more isolated, repeatable and incremental.