Azure: Measuring Code Coverage for a Cloud based Application - Introduction

To measure the code coverage for the Unit tests(s) or other BVT(s) running in the build server, we simply enable the Code Coverage settings in build definition. The Visual Studio Code Coverage tool, then will automatically analyze all the binaries that load during the testing process in the build server. But for an applications running on remote machine, we need customization w.r.t the following

1. When to instrument and inject the binaries into application package during build process.
2. A mechanism for monitoring and collecting code coverage data from the remote machine , post Test Run.
3. Reporting the coverage analysis along with Test Results.

Excluding third party tools, the two most popular choice for Code Coverage analysis tools for a .Net application are  

1. Visual studios profiling Tools - consisting of vsinstr, vsperfmon, vsperfcmd for measuring the Code Coverage.
2. Magellan - Internal to Microsoft.

If the remote machine happens to be an On-Premise VM, either one would suffice our requirement. However in case of Cloud VM, I found Magellan to be a better choice. A quick POC, revealed the following differences

	VS Profiling Tools	Magellan
	We have to explicitly start and shutdown the Vsperfcmd , before we can access the Coverage data file.	The Covermon service auto starts when Magellan is installed. No need to shutdown the service to access the Coverage data file.
Typically on an On-Premise environment, it’s easy to start and shutdown the service, as the VMs are connected in network. But for the Cloud VM that reside outside the Corp network, it’s a daunting task. We have to use Remote Power-shell.
	The coverage data file is not generated until we successfully shutdown the Vsperfcmd process.	We have another service by name Covercmd, which can be called intermittently to push the coverage details into a coverage data file, this can be done without a need to stop Covermon service.
Shutting down the Vspercmd has its own challenges. On the Cloud VM, the one running Worker Role, the vspercmd holds on to WaWorkerHost.exe. Untill we shut down this exe, we cannot shutdown the vspercmd. Force killing this exe, will do no good , as WaWorkerHost.exe being the host process for role entry point code for worker roles , the VM will enter the recycle mode and we end up losing the coverage data recorded so far. Even if we manage to get the coverage data file, before the recycle happens, the coverage file will show no coverage for worker binaries.
	Microsoft.WindowsAzure.targets ,that’s responsible for building and publishing an Azure Package , has built in framework for instrumenting Role outputs for profiling using Vsinstr	No such support exists for Magellan.
It’s very easy to incorporate the necessary support for Magellan instrumentation into “Microsoft.WindowsAzure.targets”. Moreover we can maintain a customized version of this file in our solution itself, without modifying the original file that reside in the MSBuild folder. Using a parameter (/p: CodeCoverageBuild=”True”)   switch, can instruct which version of the Target file to load.