<?xml version="1.0" encoding="utf-8"?>
<rss xmlns:atom="https://www.w3.org/2005/Atom" xmlns:sy="https://purl.org/rss/1.0/modules/syndication/" xmlns:dc="https://purl.org/dc/elements/1.1/" xmlns:content="https://purl.org/rss/1.0/modules/content/" xmlns:wfw="https://wellformedweb.org/CommentAPI/" xmlns:slash="https://purl.org/rss/1.0/modules/slash/"
  version="2.0">
  <channel>
    <title>Chris’s Performance Testing Blog</title>
    <atom:link
      href="https://docs.microsoft.com/archive/blogs/chwitt/feed.xml"
      rel="self"
      type="application/rss+xml" />
    <link>https://docs.microsoft.com/archive/blogs/chwitt/feed.xml</link>
    <description>A place to learn about performance testing</description>
    <lastBuildDate>Wed, 17 Apr 2019 03:20:38 GMT</lastBuildDate>
    <language>en-US</language>
    <sy:updatePeriod>hourly</sy:updatePeriod>
    <sy:updateFrequency>1</sy:updateFrequency>
    <item>
      <title>Self Cleaning Load Test Database</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/self-cleaning-load-test-database</link>
      <pubDate>Mon, 30 Apr 2018 10:16:58 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=395</guid>
      <description><![CDATA[I ran into an issue where I was running out of storage on the Load Test Repository DB. I knew that...]]></description>
      <content:encoded><![CDATA[<span style="margin: 0px;color: black;font-family: 'Gill Sans MT',sans-serif;font-size: 9.5pt">I ran into an issue where I was running out of storage on the Load Test Repository DB. I knew that there were a ton of old aborted runs and other runs I didn’t need. It would have taken hours to sort through them all and delete just the aborted runs because of the number of load test names that were in the results viewer. The time it takes to remove a single test was almost 5 minutes. </span><span style="margin: 0px;color: black;font-family: 'Gill Sans MT',sans-serif;font-size: 9.5pt"> </span>

<span style="margin: 0px;color: black;font-family: 'Gill Sans MT',sans-serif;font-size: 9.5pt">Here is a Stored Procedure that you can add to the Load Test Database and execute to clean up all aborted runs. Also, if an aborted run contains some description specified by the filters it will keep it. In my case I wanted to keep aborted Runs with the term TOR in the description or the word Save.  This is going to be case sensitive!</span>

<span style="margin: 0px;color: black;font-family: 'Gill Sans MT',sans-serif;font-size: 9.5pt">First Add the SP and then create a job that will run on a regular interval to keep the DB clean. In order to automate your work to save some time!</span>

<span style="margin: 0px;color: black;font-family: 'Gill Sans MT',sans-serif;font-size: 9.5pt">You can use a similar query to the one below to see what will be deleted. These are the same filters that are used in the sample exec and will represent the runs that will be deleted. </span>

-----------Sample Query to see what will be deleted and tune your filters------------

<span> </span><span>select</span><span><span style="color: #000000"> LoadTestRunID </span>from<span style="color: #000000"> dbo</span>.<span style="color: #000000">LoadTestRun</span></span>

<span>where</span><span><span style="color: #000000"> outcome </span>=<span style="color: #000000"> </span>'aborted'</span>

<span>and</span><span> </span><span>(</span><span>Description</span><span><span style="color: #000000"> </span>not<span style="color: #000000"> </span>like<span style="color: #000000"> </span>'%TOR%'<span style="color: #000000"> </span>or<span style="color: #000000"> </span>Description<span style="color: #000000"> </span>not<span style="color: #000000"> </span>like<span style="color: #000000"> </span>'%Save%');</span>

-----------End Sample -------------

Create this stored procedure to perform the cleanup.

<span> </span><span>*************Start of SP code**********</span>

<span> </span><span>/*</span><span></span>

<span>Sample Exec:</span><span></span>

<span><span style="color: #000000"> </span></span>

<span>exec dbo.Prc_DeleteLoadTestRun_Wrapper</span><span></span>

<span>       @Outcome = 'Aborted',</span><span></span>

<span>       @Desc_NotLike1 = '%TOR%',</span><span></span>

<span>       @Desc_NotLike2 = '%Save%'</span><span></span>

<span><span style="color: #000000"> </span></span>

<span>*/</span><span></span>

<span>CREATE</span><span><span style="color: #000000"> </span>PROCEDURE<span style="color: #000000"> dbo</span>.<span style="color: #000000">Prc_DeleteLoadTestRun_Wrapper</span></span>

<span>(</span><span></span>

<span><span style="color: #000000">@Outcome </span>varchar(<span style="color: #000000">100</span>),</span>

<span><span style="color: #000000">@Desc_NotLike1 </span>varchar(<span style="color: #000000">100</span>),</span>

<span><span style="color: #000000">@Desc_NotLike2 </span>varchar(<span style="color: #000000">100</span>)</span>

<span>)</span><span></span>

<span>AS</span><span></span>

<span><span style="color: #000000"> </span></span>

<span>SET</span><span><span style="color: #000000"> </span>NOCOUNT<span style="color: #000000"> </span>ON</span>

<span><span style="color: #000000"> </span></span>

<span>DECLARE</span><span><span style="color: #000000"> @i </span>int</span>

<span>DECLARE</span><span><span style="color: #000000"> @Total </span>int</span>

<span>DECLARE</span><span><span style="color: #000000"> @LoadTestRunId </span>int</span>

<span><span style="color: #000000"> </span></span>

<span>CREATE</span><span><span style="color: #000000"> </span>TABLE<span style="color: #000000"> #TEMP</span>(<span style="color: #000000">ID </span>int<span style="color: #000000"> </span>identity(<span style="color: #000000">1</span>,<span style="color: #000000">1</span>)<span style="color: #000000"> </span>NOT<span style="color: #000000"> </span>NULL,<span style="color: #000000"> LoadTestRunID </span>int<span style="color: #000000"> </span>not<span style="color: #000000"> </span>null)</span>

<span><span style="color: #000000"> </span></span>

<span>INSERT</span><span><span style="color: #000000"> </span>INTO<span style="color: #000000"> #TEMP</span>(<span style="color: #000000">LoadTestRunID</span>)</span>

<span>SELECT</span><span><span style="color: #000000"> LoadTestRunID </span></span>

<span>FROM</span><span><span style="color: #000000"> dbo</span>.<span style="color: #000000">LoadTestRun</span></span>

<span>WHERE</span><span><span style="color: #000000"> outcome </span>=<span style="color: #000000"> @Outcome</span></span>

<span>and</span><span> </span><span>(</span><span>Description</span><span><span style="color: #000000"> </span>NOT<span style="color: #000000"> </span>LIKE<span style="color: #000000"> @Desc_NotLike1 </span>or<span style="color: #000000"> </span>Description<span style="color: #000000"> </span>NOT<span style="color: #000000"> </span>LIKE<span style="color: #000000"> @Desc_NotLike2</span>);</span>

<span><span style="color: #000000"> </span></span>

<span>SELECT</span><span><span style="color: #000000"> @i </span>=<span style="color: #000000"> 1</span></span>

<span>SELECT</span><span><span style="color: #000000"> @Total </span>=<span style="color: #000000"> </span>COUNT(*)<span style="color: #000000"> </span>FROM<span style="color: #000000"> #TEMP</span></span>

<span><span style="color: #000000"> </span></span>

<span>WHILE </span><span>(</span><span><span style="color: #000000">@i </span>&lt;=<span style="color: #000000"> @Total</span>)</span>

<span>BEGIN</span><span></span>

<span><span style="color: #000000">       </span>SELECT<span style="color: #000000"> @LoadTestRunId </span>=<span style="color: #000000"> LoadTestRunId </span>FROM<span style="color: #000000"> #TEMP </span>WHERE<span style="color: #000000"> ID </span>=<span style="color: #000000"> @i</span></span>

<span><span style="color: #000000"> </span></span>

<span><span style="color: #000000">       </span>EXEC<span style="color: #000000"> dbo</span>.<span style="color: #000000">PRC_DeleteLoadTestRun</span> <span style="color: #000000">@LoadTestRunId </span>=<span style="color: #000000"> @LoadTestRunId</span></span>

<span><span style="color: #000000"> </span></span>

<span><span style="color: #000000">       </span>SELECT<span style="color: #000000"> @i </span>=<span style="color: #000000"> @i </span>+<span style="color: #000000"> 1</span></span>

<span>END</span><span></span>

<span>GO</span><span></span>

<span><span style="color: #000000"> </span></span>

<span><span style="color: #000000">*****End of SP******</span></span>

<span><span style="color: #000000"> </span></span>

<span><span style="color: #000000">Crate a Job that runs this SP:</span></span>

&nbsp;

Step 1:

Click New

[caption id="attachment_405" align="alignleft" width="875"]<img width="875" height="791" class="size-full wp-image-405" alt="Create SQL Job" src="https://msdnshared.blob.core.windows.net/media/2018/04/Pic14.jpg" /> Step 2:[/caption]

Enter a step name

Enter the type: Transact-SQL script (T-SQL)

Select the Loadtest DB

Add the command to execute with the filters that you would like to use.

<img width="876" height="794" class="alignleft size-full wp-image-415" alt="" src="https://msdnshared.blob.core.windows.net/media/2018/04/Pic21.jpg" />Step 3:

Add the advanced options:

On success actions - Quit the job reporting Success

On failure action - Quit the job reporting failure

Click OK

<img width="889" height="796" class="alignleft size-full wp-image-425" alt="" src="https://msdnshared.blob.core.windows.net/media/2018/04/Pic31.jpg" />Click on the schedules tab of your job and set the schedule you would like to use.

<img width="1424" height="799" class="alignleft size-full wp-image-435" alt="" src="https://msdnshared.blob.core.windows.net/media/2018/04/Pic41.jpg" />

&nbsp;]]></content:encoded>
    </item>
    <item>
      <title>High Value Scenarios - 4-Step Load Test</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/high-value-scenarios-4-step-load-test</link>
      <pubDate>Thu, 28 Dec 2017 13:35:55 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=296</guid>
      <description><![CDATA[High Value Scenarios consist of multiple elements that can help to describe attributes of an...]]></description>
      <content:encoded><![CDATA[High Value Scenarios consist of multiple elements that can help to describe attributes of an applications performance effectively. There are a few different scenarios that I use on a day to day basis, which provide excellent analysis points. In this post, I will attempt to explain one of the scenarios, how to build it, how to analyze the results and why it is effective.

<strong>Stair Step</strong>
My standard load test is a four-step stair step test. I use this for standard baselining as well as KPI analysis of throughput and response times.

It looks something like this:
<img width="1257" height="541" class="alignleft size-full wp-image-305" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/StairStep1.png" />
There are four stair steps each one accounting for 0.5X of the target throughput. Where X is defined as the average hourly load of the application, either projected or taken from actual production load.

Once you have X defined, use that to calculate the number of users you need using this <a href="https://blogs.msdn.microsoft.com/chwitt/2017/10/09/calculating-vusers-for-a-scenario-using-test-mix-based-on-user-pace/">previous post</a>. After you have the number of users needed for 1X you can just start with .5X and add .5X every 15 Minutes for a total test time of one hour.

<strong>Analyzing Response Times</strong>
There are a few different things that response times can do during this scenario: stay constant, increase linearly, increase logarithmically. Consistent response times tell me that the application can handel much more load than what we are running. Linear response times show that the application is starting to queue but not so much that it is past the point of failure. Logarithmic growth shows that the is substantial queuing happing somewhere in the application and the increasing response times are adding to that.

<strong>Analyzing Throughpu</strong>t
When analyzing throughput, you should see a steady linear trend. Actual throughput should increase at the same rate of scheduled throughput from the test rig.

<strong>Putting it together</strong>
To determine how the application is performing under varying degrees of load, analyze the response times and throughput of this scenario. This should explain how the application will respond in a real-world production environment with fluctuating load.

Some great places to use this type of scenario would be in predictable workload applications. It is a great baselining tool that can provide multiple points of analysis that you just can’t get from a standard consistent load scenario and prevents the data skewing associated with a max capacity scenario.

<strong>Exceptions to this scenario</strong>
This scenario will not work well for applications that have large spikes in throughput.
This scenario will not typically tell you the maximum capacity of the system (Unless the application tips over during one of the load levels)

Example 1 – Consistent KPI:
<img width="1256" height="852" class="alignleft size-full wp-image-315" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/StairStepKPI1.png" />
Example 2 – Linear KPI:
<img width="1247" height="822" class="alignleft size-full wp-image-325" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/StairStepKPI2.png" />
Example 3 – Logarithmic KPI:<img width="1260" height="844" class="alignleft size-full wp-image-326" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/StairStepKPI3.png" />

KPI – Key Performance Indicator (Response Times, Throughput, Error Rate, CPU, etc.)]]></content:encoded>
    </item>
    <item>
      <title>Great MSDN Blogs for Testers Using VSTS</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/great-msdn-blogs-for-testers-using-vsts</link>
      <pubDate>Thu, 12 Oct 2017 09:36:29 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=195</guid>
      <description><![CDATA[Hi,

I'm just going to post a bunch of redirects here to help you find other testing...]]></description>
      <content:encoded><![CDATA[Hi,

I'm just going to post a bunch of redirects here to help you find other testing resources.

&nbsp;

<a href="https://blogs.msdn.microsoft.com/devops/tag/testing/"><span style="color: #0000ff">https://blogs.msdn.microsoft.com/devops/tag/testing/</span></a>

This blog is brought to you by the team developing the testing capabilities in Visual Studio.

&nbsp;

<a href="https://blogs.msdn.microsoft.com/testingspot/"><span style="color: #0000ff">https://blogs.msdn.microsoft.com/testingspot/</span></a>

This is a blog that my team puts out monthly, it contains various performance and testing information.]]></content:encoded>
    </item>
    <item>
      <title>Four Traits of Great Performance Testers</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/four-traits-of-great-performance-testers</link>
      <pubDate>Wed, 11 Oct 2017 12:05:16 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=175</guid>
      <description><![CDATA[Whether you are a hiring manager looking for the right candidate or a current performance tester who...]]></description>
      <content:encoded><![CDATA[Whether you are a hiring manager looking for the right candidate or a current performance tester who wants to become better, these four traits will help.

<strong>Passion in technology</strong>

My first question in every interview is “What is the reason you perused a career in the IT field?”. This filters out people who are not passionate fast. It’s not really a question that someone can fake, and you will be able to tell the truly passionate apart from the rest.

One candidate, particularly sticks out in my mind as truly passionate. When asked the question, he responded that he loved computers from a young age and wanted to learn programming to improve this video gaming experience. He had no experience and was just out of college, but he never needed help, never stumbled, and always searched out the answers to solve complex problems every day. Although the job was new troubleshooting computer issues for him was not because of his passion for technology.

<strong>Curiosity</strong>

Analyzing results is not something that you can just apply a formula to and solve. A good performance tester should be curious about their results, and want to understand what is happening. Over time through this curiosity they will begin to recognize trends and see through the results what the application is doing.

None of my large projects have ever been the same, and without curiosity I would be lost. Each time you test an application there will be some components you have never worked with or business processes that you have never used. This can be perceived as terrifying for the content or the dream of a curious tester. Asking the right questions at the right time can mean the difference between success and failure.

<strong>Ability to recognize patterns</strong>

Patterns are important in testing, being able to recognize those patterns and use them to your advantage will greatly increase your ability to performance test. This first begins with scripting and follows all the way through to results analysis. Even recognizing patterns in your delivery, on boarding a new application, and results analysis will allow you to streamline your entire workflow. No two applications are the same, but the way we deliver the service may be.

<strong>Good knowledge of how IT and Business function together</strong>

<strong><em><u>Performance is about the user</u></em></strong> of the application. Every good performance tester I have met is able to reduce issues to how it will affect the end users. There will be occasions where you must become creative and determine the applications projected throughput by understanding how the business works. From knowing how a marketing campaign will affect traffic to understanding that you can’t have more volume than butts in seats make a huge difference in your testers ability to project target throughput.]]></content:encoded>
    </item>
    <item>
      <title>Dependent Request Extraction VSTS</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/dependent-request-extraction-vsts</link>
      <pubDate>Tue, 10 Oct 2017 14:13:43 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=165</guid>
      <description><![CDATA[Sometimes your extraction fails because what you are looking for is returned in a dependent request....]]></description>
      <content:encoded><![CDATA[Sometimes your extraction fails because what you are looking for is returned in a dependent request. Here is a plug-in you can use to scan dependent requests. You can adjust the below example to search the body or headers by changing “string body = e.Response<strong>.Headers</strong>.ToString();”. In the example the plugin is searching headers only.
<pre>       [Description("Extracts Text from the Response of a Dependent Request or any redirect using regex group")]

   public class FindTextInDependentRequestResponse : WebTestRequestPlugin
   {
       Match match;
       public override void PostRequest(object sender, PostRequestEventArgs e)
       {
           //Create a collection of all dependent requests
           WebTestRequestCollection dependents = e.Request.DependentRequests;
          //Gets the request that matches the filter URL and kicks off the postrequest event 
dependents
                             .Where (d =&gt; d.Url == URLFilter)
                             .Select (d =&gt; d)
                             .ToList()
                             .ForEach(d =&gt; {
                                 d.Cache = false;
                                 d.PostRequest += new EventHandler&lt;PostRequestEventArgs&gt;(this.PostDependentRequest);
                                          });
       }
       public void PostDependentRequest(object sender, PostRequestEventArgs e)
       {
           if (e.ResponseExists)
           {
               string body = e.Response.Headers.ToString();
               match = Regex.Match(body, RegexPattern, RegexOptions.Singleline);
               if (match != Match.Empty)
               {
                   //Set value to equal the matched Regex
                   string value = match.Groups[1].Value;
                   //Add context param and set to value
                   e.WebTest.Context[TargetContextParam] = value;
                   //Add comment that the context param was created and a value was added
                   e.WebTest.AddCommentToResult(
                   String.Format("Found Regex and created Param {0}",TargetContextParam)
                                           );
               }
               else
               {
               e.WebTest.AddCommentToResult(
                   String.Format("Could not find Regex to create Param {0}",TargetContextParam)
                         );
                }
           }
       }
       [Description("The name of the context parameter to store the value")]
       public string TargetContextParam { get; set; }
       [Description("The Regex Pattern")]
       public string RegexPattern { get; set; }
       [Description("The url of the target dependent request")]
       public string URLFilter { get; set; }</pre>]]></content:encoded>
    </item>
    <item>
      <title>Calculating vUsers for a Scenario Using “Test Mix Based on User Pace”</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/calculating-vusers-for-a-scenario-using-test-mix-based-on-user-pace</link>
      <pubDate>Mon, 09 Oct 2017 15:54:09 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=115</guid>
      <description><![CDATA[Load Test vUser Models in Visual Studio

Using think time for pacing

 	Test mix based on...]]></description>
      <content:encoded><![CDATA[<strong>Load Test vUser Models in Visual Studio</strong>

<strong>Using think time for pacing</strong>
<ul>
 	<li>Test mix based on sequential test order</li>
 	<li>Test mix based on number of tests started</li>
 	<li>Test mix percentage on the number of vUsers starting tests</li>
</ul>
These load test mix models will run tests as fast as possible given a set number of virtual users. The only way to control throughput is by adding think time or changing the number of virtual users. Even with those controls, a change in response times can largely affect your throughput. These pacing models should only be used for stress testing.

<strong>Automatic pacing</strong>
<ul>
 	<li>Test mix based on user pace</li>
</ul>
Allowing the load test to control the pacing based on the number of tests per hour per vUser is your best bet for baseline testing and load testing.

There are a few challenges with this load model. One of which is that you must make sure that you have a low enough number of tests per user per hour to achieve your goal. An example of having too high of a goal throughput would be setting 3600 tests per hour per user but the test’s average test time is 3 seconds.<u> It’s impossible for 1 user to run the test 3600 times an hour</u> the maximum number of tests that one user could run is 1200 per hour. To achieve your goal maybe you would set the number of vUsers to 6 and tests per hour per user to 600 which would produce the target throughput of 3600 tests per hour.

The second challenge is dealing with multiple tests in a single scenario which share the same number of vUsers. We can create a spreadsheet to help deal with this issue, but first you should do a low load test shakeout\smoke to get a starting point for response times. You will need the below information as a starting point.

<img width="229" height="105" class=" wp-image-126 alignleft" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/Ap_01.png" />

&nbsp;

&nbsp;

&nbsp;

&nbsp;

Next, we will create another column labeled “Total Time” and the calculation will be “Target Throughput” * “Average Test Time”. This will give you the total time needed in seconds for the test to run.

<img width="393" height="140" class="alignleft wp-image-135" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/Ap_02.png" />Then sum up the “Total Time” Column and divide by 3600 seconds to get the minimum number of vUsers needed. I have rounded up the number in this sheet using the Celling function.

=CEILING(SUM(G2:G6)/3600,1)

&nbsp;

&nbsp;

&nbsp;

<img width="388" height="207" class="alignleft wp-image-145" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/Ap_03.png" />

Create another input field where you can manual set the number of vUsers, and set it to a number higher than the minimum number.

Then add a column “Tests per User per Hour” and set the formula to be “Target Throughput” / “Number of vUsers”.

&nbsp;

&nbsp;

&nbsp;

&nbsp;

&nbsp;

Finally, if you want to create some validation, you can use conditional formatting to let you know the risk of running the scenario as defined. Create a <img width="368" height="292" class="wp-image-146 alignleft" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/Ap_04.png" />column named “Single vUser Time” and set the formula to be “Tests per User per Hour” * “Average Test Time”. Sum up this new column and the result should be less than 3600. Ideally it should be much less than 3600 in case response times are higher under load. Use conditional formatting on the cell where the sum exists, like the below screenshot.

&nbsp;

&nbsp;

&nbsp;

&nbsp;

&nbsp;

&nbsp;

&nbsp;

Your new vUser pacing calculator will look something like this when finished.

<img width="626" height="325" class="alignleft wp-image-116" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/Ap_05.png" />

&nbsp;]]></content:encoded>
    </item>
    <item>
      <title>Automate Your Tests for Agile</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/automate-your-tests-for-agile</link>
      <pubDate>Fri, 06 Oct 2017 13:29:55 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=96</guid>
      <description><![CDATA[The prevailing wisdom a few years back was that performance testing doesn’t fit into an Agile...]]></description>
      <content:encoded><![CDATA[<span style="color: #000000;font-family: Calibri">     The prevailing wisdom a few years back was that performance testing doesn’t fit into an Agile environment. I disagree with that idea wholeheartedly, and recognize that performance testing done correctly can work even with Agile teams. I have seen members of my team add performance tests into the CI Pipeline and perform continuous testing.</span>

<span style="color: #000000;font-family: Calibri">     <span style="font-family: Calibri">In order to achieve the velocity needed to be successful, a</span>utomate as many of the steps of your test as possible. As a performance test engineer, you are not just automating the HTTP protocol requests anymore, you need to be able to automate the test execution, set-up, and results as well. I like to use PowerShell for the execution and set-up tasks and Excel for the reporting, when working with Visual Studio Load Tests. </span>

<b><span style="color: #000000;font-family: Calibri">Test Set-up</span></b>

<span style="color: #000000;font-family: Calibri">     Automating your test set-up may include activities such as: restoring databases, recycling IIS, restarting services or machines, checking disk space on servers, setting up test agents and controllers, and maybe even running a smoke test. This should be a click and go script, meaning you can run it anytime and free yourself up to work on more important work, like new automation, test planning, results analysis. For example, if you need to reset IIS on servers prior to your test you could have a PS (PowerShell) script that looks like this. </span>
<pre><span style="color: #000000;font-family: Calibri">## restart iis on servers</span>
<span style="color: #000000;font-family: Calibri">function resetIIS($server){</span>
<span style="color: #000000;font-family: Calibri">    ## $server could be a single machine or an array of machines</span>
<span style="color: #000000;font-family: Calibri">    ## usage:</span>
<span style="color: #000000;font-family: Calibri">    ##   $server = @("web01","app01","app02")</span>
<span style="color: #000000;font-family: Calibri">    ##   resetIIS $server</span>
<span style="color: #000000;font-family: Calibri">    ## powershell command to reset iis</span>

<span style="color: #000000;font-family: Calibri">    Invoke-Command -scriptblock {iisreset} -ComputerName $server -AsJob </span>

<span style="color: #000000;font-family: Calibri">    ## using Get-Job allows multiple servers to restart iis in parallel</span>
<span style="color: #000000;font-family: Calibri">    Get-Job | Wait-Job | Format-Table</span>
<span style="color: #000000;font-family: Calibri">}</span></pre>
<span style="color: #000000;font-family: Calibri">     Something important to note with the above script, it is a function. Create all your activities as functions, that way you can reuse them quickly as needed. Highly cohesive and loosely coupled functions are the goal when deciding what to add to each function.  </span>

<span style="color: #000000;font-family: Calibri">     My recommendation is to fist create a checklist. Jot down all the things you manually do before a load test, and then start automating them one at a time. You may need help, ask your DBA’s to write you a SQL script to restore the Database, talk to your operations team about getting access to the servers to perform IIS Reset. </span>
<pre><span style="color: #000000;font-family: Calibri">$databases = @(“DB01”,”DB02”)</span>
<span style="color: #000000;font-family: Calibri">$IISservers = @("web01","app01","app02")</span>
<span style="color: #000000;font-family: Calibri">restoreDB($Databases)</span>
<span style="color: #000000;font-family: Calibri">resetIIS($IISservers)</span>
<span style="color: #000000;font-family: Calibri">executeSmokeTest()</span></pre>
<b><span style="color: #000000;font-family: Calibri">Test Execution</span></b>

<span style="color: #000000;font-family: Calibri">     If this is not your first time running a load test, don’t sit and watch it. Automate the execution and look at the results when it is finished. You may choose to automate the ability to know if the smoke test passed or you might want to add a manual verification step before the execution. Visual Studio exposes an executable that can be used to launch tests from the command line. </span>

<span style="color: #000000;font-family: Calibri">     It can be found at C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe</span>
<pre><span style="color: #000000;font-family: Calibri">switch ($VSversion) {</span>
<span style="color: #000000;font-family: Calibri">        2013 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe"}</span>
<span style="color: #000000;font-family: Calibri">        2015 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\mstest.exe"}</span>
<span style="color: #000000;font-family: Calibri">        2017 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\mstest.exe"}</span>
<span style="color: #000000;font-family: Calibri">    }</span>
<span style="color: #000000;font-family: Calibri">$mstestArguments = @($testFile, $testSetting)</span>
<span style="color: #000000;font-family: Calibri">&amp; $mstest $mstestArguments &gt;&gt; $testoutLog</span></pre>
<span style="color: #000000;font-family: Calibri">     Now that you have everything scripted; how you choose to launch it is up to you. You could take it a step further and run from the windows task scheduler, or add it to a CI/CD pipeline. Even just using the script to launch from PowerShell will help you speed up delivery. </span>

<b><span style="color: #000000;font-family: Calibri">Results</span></b>

<span style="color: #000000;font-family: Calibri">     I will be adding more articles about results analysis with specifics around the queries that I use and how to pivot the data in Excel. Let me know in the comments if this is something you are interested in. </span>

<span style="color: #000000;font-family: Calibri">     Power Query in Excel is amazing, use it. The goal is to have preformatted tables and charts that you can use to quickly determine if the test met the acceptance criteria and how your changes to code or SQL affected the Key Performance Indicators (KPIs). </span>

<span style="color: #000000;font-family: Calibri">     The next time a test runs you should just have to hit refresh on the data tab and your results will be there. Over time you will see trends and characteristics in the KPIs that will point you to where you need deeper analysis. After deep diving the data you will notice things that are missing from your automated results and then you add them in. Eventually, you will have a great automated results solution.</span>

&nbsp;]]></content:encoded>
    </item>
    <item>
      <title>Load Test Network Connection Monitoring</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/load-testing-network-connection-monitoring</link>
      <pubDate>Thu, 05 Oct 2017 11:50:14 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=85</guid>
      <description><![CDATA[Questions around how to understand the number of connections an application is using have always...]]></description>
      <content:encoded><![CDATA[Questions around how to understand the number of connections an application is using have always existed, and have become important for various reasons over the years. There was one instance that changed my perspective on monitoring active and inactive network connections. It was an application which used an Oracle backend. At the time we were seeing issues in production and it turned out that they were happening when the number of connections grew to over 10,000. Using netstat we were able to determine that a majority of the connections were not even being used and were created with the IIS connection pool in the connection strings. So, the easy resolution was to reduce the minpool size for the connection strings in the webconfig.

Using netstat is not a new idea either, but I have put together some Powershell to make it easier. This will allow you to monitor any server’s connections, then after your test is done you can dump the data into an excel spreadsheet and perform some analysis.

$ErrorActionPreference="SilentlyContinue"
$TimetoRunInMinutes = 5
$srvName = $env:computername
$outDir = 'C:\output\'
$outFile = $outDir+“NetstatReport_” + $srvName + ".txt"

for( $n = 1;$n -le $TimetoRunInMinutes; $n++)

{
$Time = Get-Date -Format G | out-File $outfile -append
netstat -an |out-file $outfile -append
# wait for a minute
Start-Sleep 60
}

&nbsp;

Have fun and let me know if you find an interesting way to use this!]]></content:encoded>
    </item>
    <item>
      <title>Adjustable Think Time For Dynamic Polling Requests</title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/adjustable-think-time-for-dynamic-polling-requests</link>
      <pubDate>Wed, 04 Oct 2017 09:07:34 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=55</guid>
      <description><![CDATA[So, you just recorded your script and see a polling request from the browser and if that isn’t...]]></description>
      <content:encoded><![CDATA[<span>     So, you just recorded your script and see a polling request from the browser and if that isn’t painful enough you also notice that the time between requests changes. Never fear, we have a plug-in for that. </span><span> </span>

<span>      </span><span>First, let’s deal with the pesky polling request by using the built-in loops and extraction rules. This is going to be different depending on your developer’s implementation of the polling, and you should observe the traces you have captured to determine a good way to know when all the data has been returned from the server. In this example the response from the polling request contains a value named Status. When Status = Idle I know that all the data has been returned and I can stop polling.</span><span>
</span><span>Let me walk you through an example using the code provided. </span><span> </span>

<span> <img width="621" height="327" class="alignleft size-full wp-image-76" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/first.png" /></span>

<span>We can see that there is a response variable named Status, which we know is Idle when the server is finished executing. </span><span>
</span><span> </span>

<span> </span>

<span> </span><span> </span>

&nbsp;

&nbsp;

&nbsp;

&nbsp;

<span><img width="300" height="183" class="alignleft size-medium wp-image-75" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/second-300x183.png" /></span>

&nbsp;

<span>So First let us create an extraction rule for Status. I decided to use Regex, and you can use your favorite extraction method. </span><span> </span>

<span> </span>

<span>
</span><span> </span><span> </span>

&nbsp;

<span><img width="300" height="171" class="alignleft size-medium wp-image-66" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/third-300x171.png" />Next, we create a loop that terminates when status = Idle. In this case we used the string comparison loop. </span><span>
</span><span> </span>

<span> </span><span> </span>

<span>
</span><span></span>

&nbsp;

&nbsp;

<span><img width="300" height="204" class="alignleft size-medium wp-image-57" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/forth-300x204.png" />From the request we notice there is a polling interval, so during the test in order to simulate the traffic generated we need to closely adhere to this polling interval. Let’s extract that value as well. </span><span>
</span><span> </span>

<span>
</span><span> </span><span> </span>

&nbsp;

&nbsp;

&nbsp;

<span>Next, we apply some test code that creates a request plugin to dynamically set the think time of a request, and allow it to use a context parameter.</span><span>
</span><span></span>
<pre><span>public class ThinkTimeFromPolling:WebTestRequestPlugin</span><span> 
</span><span>    {</span><span> 
</span><span>        [Description("The context parameter that stores the polling value")]</span><span> 
</span><span>        public string TargetParam { get; set; }</span><span> </span>
<span> 
</span><span>        public override void PreRequestDataBinding(object sender, PreRequestDataBindingEventArgs e)</span><span> 
</span><span>        {</span><span> 
</span><span>            if(e.WebTest.Context[TargetParam] != null)</span><span> 
</span><span>            {</span><span> 
</span><span>                double Thinktime = double.Parse(e.WebTest.Context[TargetParam].ToString());</span><span> 
</span><span>                e.Request.ThinkTime = Convert.ToInt32(Thinktime/1000);</span><span> 
</span><span>            }</span><span> 
</span><span>            </span><span> 
</span><span>        }</span><span> 
</span><span>    }</span><span> </span></pre>
<span>Finally, we compile then add the plugin to the request inside the loop, setting the extracted polling interval to the think time. </span><span>
</span><span> </span><span> </span><span> </span><span> </span><img width="300" height="102" class="alignleft size-medium wp-image-56" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/fifth-300x102.png" />

&nbsp;

&nbsp;

<span>Have a great day and happy testing!</span><span></span>]]></content:encoded>
    </item>
    <item>
      <title>Little's Law and How It Impacts Load Testers </title>
      <link>https://docs.microsoft.com/archive/blogs/chwitt/littles-law-and-how-it-impacts-load-testers</link>
      <pubDate>Wed, 04 Oct 2017 08:42:21 GMT</pubDate>
      <dc:creator><![CDATA[Christopher Witte]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.msdn.microsoft.com/chwitt/?p=16</guid>
      <description><![CDATA[Load testing is all about queuing, and servicing the queues. The main goals in our tests are...]]></description>
      <content:encoded><![CDATA[<span>     Load testing is all about queuing, and servicing the queues. The main goals in our tests are parts of the formula itself. For Example: the response times for a test is equivalent to service times of a queue, load balancing with multiple servers is the same as queue concurrency. Even when we look at how we are designing a test we see relationships to this simple theory. </span><span> </span>

<span>I will walk you through a few examples and hopefully open up this idea for you to use in amazing ways. </span><span> </span>

<b><span>How to determine concurrency during a load test? (Customer Question: </span></b><strong><span>What was the average number of concurrent requests during the test? What about for each server?</span>) </strong>

<b><span>Little's Law -&gt; Adjusted to fit load testing terminology</span></b><span> </span>

<span>L = λW</span><span> </span>

<span>L is the average concurrency</span><span> </span>

<span>λ is the request rate</span><span> </span>

<span>W is the Average Response Time</span><span> </span>

<span>For Example: We have a test that ran for 1 hour, the average </span><span>response times were 1.2 seconds. The average number of requests sent to the server by the test per second was 16.</span><span> </span>

<span>L = 16 * 1.2</span><span> </span>

<span>L = 19.2</span><span> </span>

<span>Average number of concurrent request processing on the servers was 19.2 averaged per second.</span><span> </span>

<span>Let’s say we have perfect load balancing, of 5 servers.</span><span> </span>

<span><img width="115" height="71" class="size-full wp-image-25 alignleft" alt="" src="https://msdnshared.blob.core.windows.net/media/2017/10/geq.png" /></span><span>M = Number of Servers/Nodes</span><span> </span>

<span>L = Concurrent requests per Server/Node</span><span> </span><span>.</span><span> </span>

<span>L= 19.2/5</span><span> </span>

<span>So, each server is processing ~3.84 concurrent requests per second on average</span><span> </span>

<span>     This explains why, when we reach our maximum concurrency on a single process, response times begin to grow and throughput levels off. Thus, we need to optimize the service times, or increase the concurrency of processing. This can be achieved by either scaling out or up depending on how the application is architected.</span><span> </span>

<span>     Also, this is not all entirely true, there will be time spent in different parts of the system, some of which have little to no concurrency, two bits cannot exist concurrently on the same wire electrically, as of writing this article.</span><span> </span>

<b><span>Can we achieve the scenario objectives given a known average</span></b><b><span> response time hypothesis?</span></b><span> </span>

<span lang="EN-US" xml:lang="EN-US">For tests that use “per user per hour pacing”, we can determine the minimum number of users necessary to achieve the target load given a known average response time under light load. You can obtain the average response times from previous tests or you can run a smoke test with light load to get it.</span>

<span> </span><span> <img width="181" height="111" class="alignleft size-full wp-image-26" alt="U = Ceil(G/(3600/W))" src="https://msdnshared.blob.core.windows.net/media/2017/10/leq.png" /></span>

3600 seconds in an hour

<span>W = Average Response</span><span> Times in seconds for the test case you are calculating</span><span> </span>

<span>Number of users needed U = 3600/W </span><span> </span>

<span>G = Throughput Goal</span><span> (Transactions per hour)</span>

<span> </span><span>     </span>

<span>     Let’s say we have a average</span><span> response time of 3.6 seconds, and our goal is to run this test case 2600 times per hour. What would be the minimum number of users to achieve this?</span><span> </span>

<span>U = ceil(2600/(3600/3.6))</span><span> </span>

<span>U=3</span><span> </span>

<span>Pacing = G / U</span><span> </span>

<span>Pacing = 866</span><span> </span>

<span>     So, the scenario would be 3 users and the test case would be set to execute 866 times per user per hour. Personally, I like to add 20% additional users to account for response time growth just in case the system slows down.</span><span> </span><span>For example: I would run with 4 vUsers at 650 Tests per User per Hour, or 5 vUsers at 520 Tests per User per Hour.</span>

<span>     There are endless possibilities to using this formula to understand and predict behavior of the system, post some examples of how you would use queueing theory in the comment section below. </span><span> </span>

<span> </span><span>Have a great day and happy testing!</span><span> </span>]]></content:encoded>
    </item>
  </channel>
</rss>