Dogfooding Visual Studio Lab Management 2010

Hello, I am Dinesh Bhat, test manager for the team that builds Visual Studio Lab Management product.  In his blog on dogfooding Visual Studio 2010 Test and Lab Management products, Amit Chatterjee talked about the team structure and how the product team is dogfooding the VS 2010 Test and Lab Management products in action. In this blog, I would elaborate more on the benefits we are getting by using VS Lab Management 2010 at work.

I am going to talk about four value propositions of the product that my team has been using and getting benefit of. These are:

1. Environments

2. Friction Free Usage and collaboration

3. Integration  with Application Lifecycle Management (ALM)

4. Reduction in cost of hardware and maintenance

Environments

As Amit writes in his blog, VS Lab Management 2010 provides a visual face to your lab. Until now, I had seen my lab in a Excel sheet and on SharePoint :-) (We use SharePoint and Excel to track machine configurations and usage.) Seeing a lab that is spread across the world is an amazing feeling for me. Now, I feel I am in better control of my hardware.

Let me throw more light on how the QA team uses Lab Management product.  Lab Management is a multi-machine setup which includes a VMM server, TFS, a test controller, a Build Controller, a client which has VS and multiple hosts that are attached to the VMM server.  The QA team needs multiple shared pods. Each shared pod typically contains one or more TFS machines, a Build controller and a test controller.  Every tester would install her/his own client and connect to this shared pod. We have two types of pods.

1. Automation pods. We have about seven of these. These pods are typically used for running Build Verification Tests (BVT) and Nightly Automation Runs (NAR)

2. Self-host pods. Every week we pick a good build and Self-host it. Self-host pods save a lot of setup time for the team and also help find bugs that normally surface with continuous usage of the product. Each self-host pod lives for two weeks.

The screenshot below shows our Automation and self-host pods on Pioneer, our dogfooding TFS. In future, we plan to get our other testing pods onboard Pioneer.

Friction free usage and collaboration

Friction free Data flow and collaboration has been one of the key themes and value propositions of Visual Studio ALM. It continues to show up in Lab Management too.  The team is getting greatly benefited in productivity using some of the collaboration features. Let me explain these in detail.

1. Self-service. As you see in Amit’s blog, VS Lab Management 2010 provides rich views of the environment list.  The team members can now create environments without depending on the lab technician. They can also look for an existing environment before creating a new one.

2. Last accessed date has been pretty useful.  When VM hosts run out of memory, people typically use this column to identify running environments which are idle and shut them down by notifying the owner

3. “Mark in use” is a feature in the product which team is using for collaboration. Testers put notes in the Mark in use edit box of the shared environment so that others aware of who is using it.

ALM integration

ALM integration is one of key value propositions of the Lab Management toolset.  As Amit’s blog  explains, we have been using the virtual environments primarily for three types of activities.

1. Manual test passes.

2. Automated test runs.

3. Rolling builds.

Let me elaborate more on our nightly automation runs. We have about 1200 automated tests in one of our features. If you run these tests one machine it would take about 14 hours. So, we created a lab environment with two VMs in it. We wrote a customized workflow to deploy nightly build of Visual Studio on these VMs. Once the builds are installed the test run starts automatically. This entire process is fully automated and now takes about 7 hours.  I should mention that the deployments logs saved us great amount of debugging effort.  In the past, we had custom deployment logs.  Every machine that had a build deployment created a log file and stored it on the local machine. In case of multi machine deployments, one had to log into every machine and analyze the logs to troubleshoot. With VS Lab Management, we now have multi machine deployment logs at one place visible to all. The logs are organized with indentation and also point to the errors with icons. Now it takes no time to troubleshoot the deployment failures. Moreover, the deployment logs now have “Time spent” in each section of the deployment. This helps the team to identify long running tasks and optimize the deployment time.

Cost of ownership and maintenance

Finally, this is about how visualization and Lab Management helped us to cut total cost of ownership for the lab hardware.

In 2007, my lab had 72 Dell OptiPlex Gx 620 boxes. Each box had 2 GB of RAM.  Since we were not using virtualization, each box was acting as one machine.  I used to pay $12 per box per month for maintenance. It used to take anywhere from 8 to 24 hours to reimage a box and get it ready for testing.

In mid-2008, we decided to adopt virtualization. We bought 20 HP Compaq DC 7800 boxes. Each box had 4 GB RAM. Each tester owned one box. We used an internal tool call “Virtual Lab” which operates on top of Microsoft Hyper-V.   With this, we got about 20% increase in tester productivity.  (Internally, we use a metric called “Time to test”. This metric represents the actual time a tester gets to do testing. This excludes time lost in preparing machines, installing builds, troubleshooting etc.). Although using virtualization helped us a lot in improving tester productivity, we were not optimally utilizing the hardware resources. In addition, it was hard to collaborate on the machine resources within the team as this was not a centrally managed system.

In October 2009, we started to dog food VS Lab Management 2010.  We took away the some of the Hyper-V boxes used by the testers and created a Host group. We added three 32 GB boxes to take the total number of hosts to 12.

Below table depicts the overall cost savings.

 

*at the rate of $12/machine/month

 

Overall, using VS Lab Management 2010 has been a great excitement for us. In future, my team plans to exercise other value propositions, such as, environment templates, stored environments and network isolation.  I will keep you posted on our learning from that.

Dinesh Bhat, Test Manager