Tales from the field

Load Testing with Agents running on Windows Azure – part 3

rlfh — Fri, 08 Apr 2011 01:03:00 GMT

** Important note – Please read before reading further **

The content in this post was first written in 2011, exploring the earlier versions of Windows Azure. Since then, Windows Azure has evolved quite a lot, as well as other services in the Microsoft portfolio.

We now have an official solution to run load tests on Azure - load testing is part of Team Foundation Service and a preview is available. If you haven't tried it already, I'd encourage you to try it.

Find out more at https://aka.ms/loadtfs.

If you’ve read the last 2 posts, you already know we’re building a proof-of-concept test rig using Visual Studio Test Agents hosted on Windows Azure. For the final part of this post, we’ll look at the deployment procedure.

I usually start by deploying just 1 instance from Visual Studio. There’s no point in jumping to the deployment of the full rig because currently there’s no option to setup Windows Azure Connect other than using the Management Portal and doing it manually. Let’s go by it step-by-step.

To deploy the first instance, the number of instances of my Worker Role in the Service Configuration file should be set to 1.

You can then Publish by right-clicking on the Cloud project and choosing Publish. Configure your target Hosted Service and environment and click OK to start deploying. At this point, Visual Studio will compile and generate your service package and then start uploading it to Windows Azure. Afterwards, it will trigger the deployment of the configured instance for your role. The whole process might take a while (around 15 minutes).

During this time, you should navigate to the Management Portal and take a look at what’s going on from the Hosted Services screen. The Hosted Services screen will allow you to track the deployment process.

The other screen you should be looking at periodically is the Virtual Network screen. When the instances are deployed, because they are enabled for Connect, they will show up here on the section “Activated Endpoints”. This section will contain a reference to your Controller machine, since you installed the Connect agent there.

Once it shows your new instance (a new entry named RD-something), you’ll be able to navigate to the “Groups and Roles” section, create a Group and create a Role Connection between your service and the Endpoint Group you just created.

It will look something like this:

Once you’ve done this, you can sit and wait for this first instance to install. You’ll notice that, on the Controller machine, once the instance is set as ready, you’ll be able to get an IPv6 for it and the Connect agent icon changes to show the connection is active.

If you feel the deployment is hung, a good tip is to reimage the instance.

Once this instance is Active, you can just change the configuration to add some instances. You can use the Management Portal to do so, choosing the Deployment and clicking “Configure” on the ribbon. Try changing the number of instances to 5, for example.

You’ll notice that every new instance you create will register with your local test Controller and become an Agent itself.

The only thing left is to reimage your first image so that it too registers with the Controller and your rig is now done! You should now be able to run Test Projects using this rig and collect the results.

You can download a sample project here. This sample project, the scripts and the code are provided as-is, without responsibility for support, and is the result of my curiosity and very limited testing. Remember that you need to change some settings on the configuration and the scripts, such as the download URL for the VS Agent software and the usernames and passwords which are created or used in the schedule task, as well as the name of your Controller Machine on the VS Agent setup command line.

Load Testing with Agents running on Windows Azure – part 2

rlfh — Fri, 08 Apr 2011 01:02:00 GMT

** Important note – Please read before reading further **

We now have an official solution to run load tests on Azure - load testing is part of Team Foundation Service and a preview is available. If you haven't tried it already, I'd encourage you to try it.

Find out more at https://aka.ms/loadtfs.

If you’ve read the last post, you know by now that we’re building a Load Test rig hosted on Windows Azure using Visual Studio 2010. Let’s take a look at our 2 different components.

Controller

For the Controller machine, you need Visual Studio 2010 and the Visual Studio 2010 agents, so that you can install the Controller agent.

You should create a user account vstestagent that has the same password as the vstestagent user created in the setup.cmd on the Agent machines. This is required because the machines are not domain-joined but must authenticate with each other.

You must also install the Endpoint software for Windows Azure connect. You can do this from the Management Portal by clicking “Install Local Endpoint” on the ribbon.

There’s really not more you need to do or install on this machine.

Agent

My immediate idea was to build a custom VM for this role, but a colleague pointed me to right direction and suggested using worker roles and some startup tasks to deploy the needed software.

I started by creating a Cloud project in Visual Studio and adding a Worker Role. The code for the worker role is rather basic: in fact, the worker role does absolutely nothing because all of the work is done in the startup tasks.

using System;

using System.Collections.Generic;

using System.Diagnostics;

using System.Linq;

using System.Net;

using System.Threading;

using Microsoft.WindowsAzure;

using Microsoft.WindowsAzure.Diagnostics;

using Microsoft.WindowsAzure.ServiceRuntime;

using Microsoft.WindowsAzure.StorageClient;

 

namespace WorkerRole1

{

    public class WorkerRole : RoleEntryPoint

    {

        public override void Run()

        {

            // This is a sample worker implementation. Replace with your logic.

            Trace.WriteLine("WorkerRole1 entry point called", "Information");

 

            while (true)

            {

                Thread.Sleep(10000);

                Trace.WriteLine("Working", "Information");

            }

        }

 

        public override bool OnStart()

        {

            // Set the maximum number of concurrent connections 

            ServicePointManager.DefaultConnectionLimit = 12;

 

            // For information on handling configuration changes

            // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.

 

            return base.OnStart();

        }

    }

}

The service definition includes the Worker Role, loads required modules, including Connect, and defines the startup tasks that must run during startup.

xml version="1.0" encoding="utf-8"?>

<ServiceDefinition name="TestAgent" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">

  <WorkerRole name="WorkerRole1" vmsize="Medium">

    <Imports>

      <Import moduleName="Diagnostics" />

      <Import moduleName="Connect" />

      <Import moduleName="RemoteAccess" />

      <Import moduleName="RemoteForwarder" />

    Imports>

    <Startup>

      <Task commandLine="setup.cmd" executionContext="elevated" taskType="simple" />

      <Task commandLine="setupfw.cmd" executionContext="elevated" taskType="simple" />

    Startup>

  WorkerRole>

ServiceDefinition>

The 2 tasks declared will be responsible for: 1) configuring the software and 2) configure the firewall rules required for the Controller-Agent connection.

To configure an Agent, the following tasks will have to performed:

So, what do the scripts look like? Let’s start by looking at setup.cmd, which is more or less self explaining:

powershell -command "set-executionpolicy Unrestricted"

powershell -command ".\setup.ps1" -NonInteractive >; out.txt

 

unzip -q -o "c:\Resources\temp\VSAgent\VSAgent.zip" -d "c:\Resources\temp\vsagentsetup"

c:\Resources\temp\vsagentsetup\testagent\setup.exe /q /f /norestart /unattendfile e:\approot\setupagent.ini

 

net user vstestagent yourpasswordgoeshere /add

net localgroup Administrators vstestagent /add

 

REM Create a task that will run with full network privileges.

net start Schedule

 

schtasks /CREATE /TN "Configure Test Agent Service" /SC ONCE /SD 01/01/2020 /ST 00:00:00 /RL HIGHEST /RU admin /RP youradminpasswordgoeshere /TR e:\approot\configagent.cmd /F

 

schtasks /RUN /TN "Configure Test Agent Service"

The only tricky bit in the script above is the fact that you need to configure a scheduled task that runs immediately to actually run a command line using credentials other than the system account.

The setupfw.cmd script is nothing more than a set of commands to open the common RPC/WMI/etc ports, plus the Agent Service default port on port 6901. Note that I tested this configuration only and I was not concerned about creating a least privilege or closed setup. This may need some tuning to be more secure.

netsh advfirewall firewall add rule name="RPC" dir=in action=allow service=any enable=yes profile=any localport=rpc protocol=tcp

netsh advfirewall firewall add rule name="RPC-EP" dir=in action=allow service=any enable=yes profile=any localport=rpc-epmap protocol=tcp

 

netsh advfirewall firewall add rule name="Port 135 TCP" dir=in action=allow service=any enable=yes profile=any localport=135 protocol=tcp

netsh advfirewall firewall add rule name="Port 135 UDP" dir=in action=allow service=any enable=yes profile=any localport=135 protocol=udp

netsh advfirewall firewall add rule name="Port 136 TCP" dir=in action=allow service=any enable=yes profile=any localport=136 protocol=tcp

netsh advfirewall firewall add rule name="Port 136 UDP" dir=in action=allow service=any enable=yes profile=any localport=136 protocol=udp

netsh advfirewall firewall add rule name="Port 137 TCP" dir=in action=allow service=any enable=yes profile=any localport=137 protocol=tcp

netsh advfirewall firewall add rule name="Port 137 UDP" dir=in action=allow service=any enable=yes profile=any localport=137 protocol=udp

netsh advfirewall firewall add rule name="Port 138 TCP" dir=in action=allow service=any enable=yes profile=any localport=138 protocol=tcp

netsh advfirewall firewall add rule name="Port 138 UDP" dir=in action=allow service=any enable=yes profile=any localport=138 protocol=udp

netsh advfirewall firewall add rule name="Port 139 TCP" dir=in action=allow service=any enable=yes profile=any localport=139 protocol=tcp

netsh advfirewall firewall add rule name="Port 139 UDP" dir=in action=allow service=any enable=yes profile=any localport=139 protocol=udp

 

netsh advfirewall firewall add rule name="Port 445 TCP" dir=in action=allow service=any enable=yes profile=any localport=445 protocol=tcp

netsh advfirewall firewall add rule name="Port 445 UDP" dir=in action=allow service=any enable=yes profile=any localport=445 protocol=udp

 

netsh advfirewall firewall add rule name="Port 6901 TCP" dir=in action=allow service=any enable=yes profile=any localport=6901 protocol=tcp

netsh advfirewall firewall add rule name="Port 6901 UDP" dir=in action=allow service=any enable=yes profile=any localport=6901 protocol=udp

 

netsh advfirewall firewall add rule name="Port 6910 TCP" dir=in action=allow service=any enable=yes profile=any localport=6910 protocol=tcp

netsh advfirewall firewall add rule name="Port 6910 UDP" dir=in action=allow service=any enable=yes profile=any localport=6910 protocol=udp



Now that the scripts are ready, how is this deployed to Windows Azure? Let’s talk about that in the next part of the post.

Load Testing with Agents running on Windows Azure – part 1

rlfh — Fri, 08 Apr 2011 01:00:00 GMT

** Important note – Please read before reading further **

We now have an official solution to run load tests on Azure - load testing is part of Team Foundation Service and a preview is available. If you haven't tried it already, I'd encourage you to try it.

Find out more at https://aka.ms/loadtfs.

I’ve been following the cloud computing scene with some interest as I find cloud computing to be quite an interesting approach to IT for some/many companies. There are a few characteristics of cloud computing that I find very useful (and why not, exciting!), in particular the concepts around elasticity, when you scale to meet demand without worrying about hardware constraints, and the pay-as-you-go model, so that you are only charged when you actually deploy and use the system.

Of course a big part of getting value from cloud computing is identifying the most adequate workloads to take to the cloud, so after some thinking (and based on some needs from a customer) I’ve decided to try to make a Load Testing rig where the Agents are running on Windows Azure. This scenario has some aspects that make it “cloudable”:

Running the Agents as Windows Azure instances allows me to quickly create more of them and, thus, generate as much load as I can;
A load test rig is not something which is used 24x7, so it’d be nice to just destroy the rig and not having it around if not needed.

So, my first step was to decide how to design this and where to place each component (cloud or on-premises) to test a more or less typical application. Based on some decisions I’ll explain later in the post, I came up with this design:

The major components are:

Agents – As intended, they should be running on Windows Azure so I could make the most of the cloud’s elasticity;
Controller – I took some time deciding on where the controller should be located: cloud or on-premises. The Controller plays an important part in the testing process as it must communicate with both the agents, to send them work to do and collect their performance data, and the system being tested, to collect performance data. Because of this, some sort of cloud/on-premises communication would be needed. I decided to keep the controller on-premises mostly because the other option would imply having to either open the firewalls to allow connection between a Controller on the Cloud and, for example, a Database server or to deploy the Windows Azure Connect plugin to the database server itself. For the customer, I was thinking of this was not feasible.
Windows Azure Connect – The Windows Azure Connect endpoint software must be active on all Azure instances and on the Controller machine as well. This allows IP connectivity between them and, given that the firewall is properly configured, allows the Controller to send work loads to the agents. In parallel, and using the LAN, the Controller will collect the performance data on the stressed systems, using the traditional WMI mechanisms.

Once I knew what I wanted to design, my next step was to start working hands-on on the scripts that would enable this to be built. My ultimate goal was to have something fully automated, I wanted as less manual work as possible. I also knew I had to make a decision about the type of role to be used for the Agent instances: web, worker or VM.

I’ll explore these topics in the next parts of this post.

Documentation fine print #1 – ViewFormPagesLockdown

rlfh — Wed, 07 Jul 2010 06:05:35 GMT

(this is the first of a series of posts to document and discuss problems that come up from not noticing a detail in the documentation)

One of my customers was trying to use the ViewFormPagesLock

Technorati Tags: sharepoint documentation configuration planning architecture

down feature in SharePoint 2007 to restrict access from anonymous users to the form pages which can be used to list items or view them individually.

This is often a good practice for external anonymous access and is a technique described in the “Plan security for an external anonymous access environment” chapter from the Planning and Architecture for Office SharePoint 2007 technical library on TechNet.

The customer had activated the feature as described in the procedure published on the ECM team blog, but things were not working as expected as the anonymous users still had access to pages like /Forms/AllItems.aspx.

Re-reading the post, I noticed a small detail right after the command. It read:

This will enable the lockdown feature on your site collection. If you already had anonymous access enabled, you'll need to go disable it, then enable it again.

Because this was placed after the command, I figured that the customer may have missed so I suggested they’d give a try and voilá!

Lesson learned: the anonymous access to a site collection needs to be reset (disabled and enabled) after the ViewFormPagesLockdown feature is enabled.

Reflections on troubleshooting

rlfh — Thu, 24 Jun 2010 05:43:02 GMT

I spend most of my work time troubleshooting or managing the process of troubleshooting technical issues onsite, with the customer teams and with Microsoft’s support people. Doing that so often, I’m found wondering on what knowledge can I take from each problem to the next and how can that be organized and used to improve how efficiently my approach to the problem is.

Don’t get me wrong, I’m not talking only about technical knowledge. My goal for this post is to reflect on how is the troubleshooting process usually done, the different approaches and the pros and cons of each, and also the different steps in each approach.

I’m not really following any methodology or have read any literature about this topic, I’m just venting and putting out some of the experiences I’ve had in the past.

“Houston, we have a problem.”

Something similar to this is frequently heard at 9am or 5pm. I’m not sure if there is a statistical correlation between time of the day and problem occurrence, but I do know that the person carrying the message will most likely:

a) not have any information about the problem, except for the fact that he/she has one;

b) not tell you the correct information about the problem;

c) include several problems in one;

d) be very, very nervous or under pressure.

Points a), b) and c) all relate to the way the problem is characterized and scoped, in the next section. Point d) will need some management skills to keep the information flowing and prevent everyone involved from panic. Let’s think about each one of these topics in detail.

Detailing the issue

Gathering and organizing the information is one of the most critical steps in the whole process. Your very first step is to find out what is, in fact, the issue. Some crucial question arise here such as:

- What is the impact of issue?

- Does it affect every user?

- What symptoms are you seeing?

- Is it happening all the time?

- Is it happening now?

- Has it happened before? How did we fix it back then? (a good knowledge base is helpful here)

- Was there some change to the system lately? (a good change management process is helpful here)

- What actions have you already taken?

- Is your goal a root-cause analysis or just restoring the service? (setting the expectations with this question is important and may save you from some political problems later)

I find myself faced with systems I don’t know anything about, so identifying the architecture pieces and the way the problematic feature works with each is relevant. This will give you an idea of the machines, products, technologies and dependencies involved and may even give you some hints on the cause of the problem. You don’t need to go into the details of every machine, but typical information such as OS, patch level and the way the information flows between systems (identifying who is the client and server and the protocols involved) should be included.

If the problem is affecting just some of the different components of the system, breaking it down and defining the dependencies will allow you to choose which area to check first.

If you’re handling a really critical situation, then point d) above comes into play and you should worry about communication. Try to keep everyone aligned for example by drawing the picture of the system and listing the symptoms on a board. This is probably not the best time to stop and write an e-mail unless you have someone who can help you with that.

Defining an action plan

Assuming that you’ve got the information above and you trust it to be true, you should have a clearer picture of what are the users experiencing which is not the normal behavior of the system. Your next step, then, is to set up an action plan to start attacking the problem.

The action plan must start with a clear goal, i.e, what are you trying to gather, prove or fix, and then include the steps that will be taken to achieve that goal. Remember the communication concerns we mentioned before? Well, if you keep everyone up-to-date and aligned in terms of the actions that will be performed and the goals that we want to achieve, it will probably help reduce the tension and make everyone work together. Note that there may not be a single possible action plan nor will your first goal be "solve the issue”. After all, if you knew the cause of the problem, it would be easy to fix it, right?

If your plan has the goal of solving the issue, for example through some configuration change, how can you measure if the issue was solved? What are you metrics? Hard and factual data is relevant here, because it will prove the issue is gone.

There are different approaches to investigating a problem. If you’re really comfortable about the technology or system and you suspect some part of it to be the root cause, you could take a bottom-up approach and start by going really deep on one or only a few components. If your gut feeling is right, then this will be a fast path to success, but if it’s wrong it could prove to be a waste of time. Being able to do this takes some experience with the technology and a good set of similar problems in your luggage.

The opposite approach is obviously the top-down approach. It starts by taking a look at indicators and monitoring from the systems at a very high level and try to dig deeper when you find something strange. If you don’t know where to start, this might be a good way to organize the work, but it may take you some time to actually get to the core of things and be able to find out interesting information.

Another possible way to plan your work is something I find quite similar to binary searching. Like in the algorithm, try to divide the system in half (or at least in large enough parts) and work top-down on each one. If you find that a certain part of the system is working correctly then you can rule them out completely and focus on the other side. If you can isolate some components, then it’s easier to choose to ones you want to dig into. I try using this pattern when I’m troubleshooting because I feel it’s usually more effective. It requires a good knowledge of the way different components interface and communicate, especially transport and data protocols, and which monitoring and logging data is associated with the interface mechanisms.

Regardless of your chosen approach, follow your action plan, gather the information you needed and proceed to the next step.

Evaluate your last move

Once you’ve achieved the goal established for the action plan, it’s time to evaluate what was done and the information that was collected. Here, you need to recall the goals you established for your action plan. Were they accomplished?

If your plans were to gather more information, then what does that information show? Does it point to a suspect or is everything looking normal? If you were planning to change some setting and fix the problem, did it work?

Start over or celebrate!

Troubleshooting, much like the scientific method, is an iterative task and it may require you to go through the last 2 steps several times before you understand what’s going on.

If your action plan did not work or if only showed the need for further actions, plan them the same way you did before and set the wheels in motion. If your problem is gone, then your work is done.

At this point, it may be a good idea to debrief and discuss what happened and to work proactively on how the problem can be prevented or at least predicted. This way, you’ll learn something from the situation that can potentially save time, money and the day later on.

These were my thoughts on managing troubleshooting processes based on my personal experience. Feel free to comment and share yours.

How to use fiddler and HTTP replay to have an offline copy of your site

rlfh — Tue, 22 Jun 2010 03:37:00 GMT

Very often, I’m asked to troubleshoot some client-side issue regarding web applications. Because, most of those times, I’m not onsite at the customer experiencing the issue, or because often this issue occurs on some remote site or branch office, it can sometimes be tough to replicate the problem or to gather enough and relevant information to troubleshoot or to try some real-time debugging techniques.

A very interesting way to achieve this is by using a combination of a very well-known tool, Fiddler, and a somewhat unknown tool called Http Replay. With these tools, we’ll work on creating an offline mirror of the browsing experience of a site, that can easily be packaged and sent to a developer for a more detailed look.

We’ll start by downloading and installing the tools. Just go here and here and get the bits to your machine. Then, execute both installers to setup the applications.

Our next step is to gather a Fiddler trace. This is something most web application developers are used to, so it should be familiar to you. Just fire up Fiddler and navigate to your desired web site.

The goal here is to visit every page and resource that you want to make available offline, so be sure to clear your browser cache before start collecting data so that all images and javascript files, which are normally cached are actually loaded and, therefore, show up on the Fiddler trace.

Here’s a sample of Fiddler after tracing www.microsoft.com:

Now, create a trace file, going to File, Save All Sessions and storing the saz file somewhere on your hard drive. This file holds all the necessary information for HTTP Replay to mimic www.microsoft.com offline, which is what we’ll do next.

HTTP Replay works as an HTTP proxy that either returns the stored response for an HTTP request it knows (because it’s stored on some trace file) or forwards the request to the actual server, if available.

To replay from a Fiddler log, you’ll need to:

- rename the "SAZ" file to "ZIP"

- extract the "raw" directory

- run HTTPREPLAY.CMD passing the "raw" directory as the logfilename

To start HTTP Replay, open a command prompt and cd to C:\Program Files\HTTPREPLAY or C:\Program Files (x86)\HTTPREPLAY on an x64 system. Next, execute a command similar to HTTPREPLAY.CMD , for example, HTTPREPLAY.CMD C:\temp\raw.

This is will fire up a browser that shows the requests contained in the trace file provided. It should be similar to this:

Note that these are the same requests that the Fiddler trace contained and that the screenshot above showed.

Now, just follow the instructions on the page: clear your browser cache, change the proxy to localhost:81 and click on the first link. For each request, the command prompt running HTTP Replay will show some output like this:

23:30:39:040 #0 - New connection accepted (127.0.0.1:64308)

23:30:39:083 #0 - 127.0.0.1:64308 -> :81 (2691 bytes / total : 2691 bytes)

23:30:39:084 #0 - GET http://www.microsoft.com/ [FOUND]

Each time a request is FOUND, it is returned from the trace file which is disconnected from the server. With this tool and the above steps, you can now replicate your site’s behavior and troubleshoot client-side easily.

As a last note, there are some interesting settings that you can use for HTTP Replay, the most interesting of which is the RESPECTTIMINGS option which makes HTTP Replay take as much time as the original request took to serve the response. However, for this you can’t use a Fiddler trace and you must use another tool called STRACE.