I was just going through my email and came across a question on one of our discussion aliases that asked if it was possible to deliver an ALM assessment for a customer without worrying about the tool set they currently use. The customer wanted to have the assessment focus on their process. The person asking the question wanted to make sure we could conduct an assessment in that manner. At first, I was surprised that the question had been asked since I figured the answer is so obvious (after all, this was an internal Microsoft discussion alias). Then it hit me. The answer is obvious to those of us who perform the assessments, and do the training, and work with dozens of different customers each year. But the general IT industry often looks at things differently. The question on the alias was:

“I need someone who can do the ALM assessment where the delivery is tools agnostic”

One of my teammates replied:

“Indeed the ALM assessment doesn't even care about which tools you use.”

I replied to his answer as follows:

“I would take it one step further and state that any ALM assessment that does focus on tooling is done incorrectly. ALM is a process, the tools are…..well, tools. The process should always dictate the tools. The tools should NEVER dictate the process.” Now I do believe the tools SHOULD SUPPORT and BOLSTER the process, but that is not the same as dictating the process.

The Process should come first

I bring this up in my blog because I find myself repeating this sentiment all the time. I delivered an ALM Testing workshop a couple weeks ago (a 3 day high level class that focuses on several different aspects of testing within an ALM based shop). I found myself repeating this message 3 or 4 times per day in response to questions that the attendees were asking. I use this message when I am teaching ALM. I use this message when I am doing an ALM assessment. I use this message when I am delivering a Testing Readiness Assessment. I use this message when I am developing a Performance Test Plan (see this post). I can use this message just about anywhere that I am planning or assessing (unless the assessment is something like “Assess how well we are taking advantage of TOOLSET X, which would be a customized offering and not subject to normal rules).

The simple message that I want to make sure I convey is that “The process should define the tools. The tools should NEVER drive the process.” In other words, you should:

1. Define your process
2. Pick tools that fit your process.
3. Then enter into a lifecycle maintenance loop:
   1. Evaluate the process/tools feedback loop
   2. Is the process still relevant?
   3. Are the tools supporting the process?
   4. Lather, rinse, repeat.

Don’t Ignore the Tools

One customer I was recently working with had different tools for source control, development and project management. In order for them to add code reviews to their workflow, they had to create a shelf set, then go to the source control tool to request a review. Then they had to manually add a task to the project management. All of these had to be updated separately and manually upon completion. They were actually avoiding doing regular code reviews because of “lack of time” in their WIP estimates. I showed them the feature of code reviews that is integrated into TFS and Visual Studio [if you have pending changes, you can simply click on the “actions” dropdown, select “Request Review,” and choose a person from the list. That’s it. TFS will make the shelf-set, create a work item task and assign the task to the person.

This is a good example of considering the tools and using them to drive EFFICIENCY. The customer can definitely benefit from the way TFS handles code review workflow. HOWEVER, it is important to know that even if they do not choose to start using TFS, they are still going to be enforcing code reviews as part of their process.

NOTE1: I have included a sproc in this post that you can download and run against a LoadTest2010 database. It will add all of the below info to the DB.

NOTE2: The last two sprocs will NOT install properly on any version of SQL Server older than 2012. If you experience any errors running the code, make sure your version of SQL is at least 2012.

About the sample code

The code here is all a work in progress. Please play with it, use it, test it and provide me feedback. At the end of this series, I will post the entire SQL codeset with any updates and fixes that get reported.

Data Warehousing

The idea of creating Data Warehouses is nothing new, and strictly speaking I do not know if what I am creating is officially a data warehouse. However, the idea fits so that’s what I am calling it. I add a few tables to the LoadTest2010 database and then populate them with aggregated and recalculated data. The beauty of the data is that it can all be gotten from existing data in the DB and does not break the current schema in any way. I will explain the table schemas first, then I will explain the stored procedures I use to populate the data into the tables.

The Tables

• VSLT_LoadTestCoreInfoThis table contains a single row for each LoadTestRun stored in the database. The main purpose is to extend the data stored in the LoadTestRun table. I will not show all of the columns in this table. You can see the columns in the SQL code and the values are self explanitory. I will explain the FLAG columns though. These are used by my front end reporting tool and help the tool decide what functionality should be exposed for any given run:
  • FLAG_ContainsPages: TRUE if the run contains web page requests.
  • FLAG_ContainsTransactions: TRUE if the run contains Transaction Timers
  • FLAG_ContainsDetailedTimingInfo: TRUE if the test run contains the actual details for any of the three 'LoadTest***Detail' tables
  • FLAG_ContainsStepLoadProfile: TRUE if the test run contains more than 1 constant value for the "User Load" counter.
  • FLAG_TestAborted: TRUE if the Outcome column of the LoadTestRun is equal to 'Aborted'.
  • FLAG_LoadTestParsingFailed: This is set to TRUE if any type of error or exception occurs during the execution of the 'VSLT_prc_Update***' stored procedures
  • FLAG_LoadTestIntervalProfile: Contains the value associated with the interval profile from the next table below.

• VSLT_IntervalProfileTypes[Normalization Table] Contains the list of interval types that could exist in a given load test run. There are two columns: The ID and Description for each type
  • 'Data May not be valid': Indicates that the routines I use to calculate profiles did not have a proper checksum
  • 'use ENTIRE RUN': Indicates that there was only a single User Load for the entire run (including any warmup)
  • 'use INTERVAL VALUES (only one interval exists)': This run contains a warmup, or a steady state step ramp, then goes constant, so there is only one interval stored for the run. Use it.
  • 'Indeterminant, multiple intervals exist': This run contains multiple valid intervals. Use the full set of interval tools to process it.

• VSLT_LoadTestIntervalsThis table contains the data about any of the constant load "intervals" discovered in a run. There is one row of data for each interval.
  • LoadTestRunId: What run is this interval associated with.
  • StartOffset: How many seconds from the beginning of the run does this constant interval start
  • EndOffset: How many seconds from the beginning of the run does this constant interval end
  • UserLoad: How many users were executing tests at this interval
  • NumberOfDataPoints: How many perfmon collection intervals occurred during this interval (if the test's sample rate is every 5 seconds and the interval is 60 seconds long, then there will be 12 data points

• VSLT_ScoreCardMachineCPU This table contains a single row of data per machine per run for any machine that had Perfmon CPU info collected. It is used to easily build summary and comparison reports. It is based on the Perfmon counter Processor\(_Total)\% Processor Time
  • LoadTestRunId: What run is this data associated with.
  • MachineName: The Computer Name
  • MachineTypeId: The Id for the type of machine this is. This field is currently not used, but exists to allow for aggregating data based on the roles of the machines
  • InstanceId: This is the Id contained in the LoadTestPerformanceCounterSample table and allows you to easily pull all data points for building charts or aggregating data
  • Qty: The number of perfmon samples collected from this machine during this run.
  • Min: The aggregated Minimum value from the perfmon data.
  • Avg: The aggregated Avg value from the perfmon data.
  • Max: The aggregated Maximum value from the perfmon data.

• VSLT_ScoreCardMachineAvailableMB This table contains a single row of data per machine per run for any machine that had Perfmon Memory info collected. It is used to easily build summary and comparison reports. It is based on the Perfmon counter Memory\Available MBytes
  • LoadTestRunId: What run is this data associated with.
  • MachineName: The Computer Name
  • MachineTypeId: The Id for the type of machine this is. This field is currently not used, but exists to allow for aggregating data based on the roles of the machines
  • InstanceId: This is the Id contained in the LoadTestPerformanceCounterSample table and allows you to easily pull all data points for building charts or aggregating data
  • Qty: The number of perfmon samples collected from this machine during this run.
  • Min: The aggregated Minimum value from the perfmon data.
  • Avg: The aggregated Avg value from the perfmon data.
  • Max: The aggregated Maximum value from the perfmon data.

• TSL_DbChangeLogsThis table tracks any changes that my stored procedures makes to the DB
  • LoadTestRunId: What run is this data associated with.
  • LogId: This is an Id for each entry in the table. This Id PLUS the LoadTestRunId form a unique key.
  • ChangeTypeId: This Id is used to map to the TSL_DbChangeLogTypes table to describe what type of change occurred.
  • ChangeTime: The timestamp when the change occurred.
  • LogComment: Any additional comments about the change will get stored here.

• TSL_DbChangeLogTypes This table holds the descriptions for the types of changes that can be made to the DB by my sprocs. Valid values (shown as [ID], '[description]') are:
  • 1, 'VSLT_prc_UpdateLoadTestCoreInfoTable'
  • 2, 'VSLT_prc_UpdateScoreCardMachineCpuTable'
  • 3, 'VSLT_prc_UpdateScoreCardMachineAvailableMBTable'
  • 4, 'VSLT_prc_GetStartingValuesForImportingPerfmon'
  • 5, 'VSLT_prc_UpdatePerfmonTablesFromImport'
  • 6, 'VSLT_prc_UpdateLoadTestIntervalTable'
  • 7, 'VSLT_prc_UpdateLoadTestCoreInfoTable_IntervalProfileFlag'

The Update Sprocs

These stored procedures are used to update the values in the above tables. There are a few things to note about these sprocs and their usage:

• These sprocs use CURSORS to go through all load test runs and only process the runs that have not already been processed. It is safe to run these sprocs as often as you would like.
• The main sproc in this group will call ALL OTHER sprocs in the group so that you can get a full update from only a single command.
• These sprocs will update the ChangeLogs info every time a change is made.

VSLT_prc_UpdateLoadTestCoreInfoTable - updates all of the data for the VSLT_LoadTestCoreInfo table. It also calls the two below sprocs.

VSLT_prc_UpdateScoreCardMachineAvailableMBTable updates all of the data for the VSLT_ScoreCardMachineAvailableMB table.

VSLT_prc_UpdateScoreCardMachineCpuTable updates all of the data for the VSLT_ScoreCardMachineCPU table.

VSLT_prc_UpdateLoadTestIntervalTable updates all of the info for the VSLT_LoadTestIntervals table. It also calls the below sproc.

VSLT_prc_UpdateCoreInfoWithIntervalFlags updates the VSLT_LoadTestCoreInfo table with the following values:

• FLAG_LoadTestIntervalProfile
• MainIntervalStart
• MainIntervalEnd

Automatically Updating Data

the following two built in sprocs can be safely modified using the attached ALTER PROCEDURE. This will allow Visual Studio to automatically update all of the VSLT data warehousing tables at the end of every run. I added a brief description below, but study the download to fully understand the changes I make. (NOTE: I am working on a routine that will automatically handle interval data on LoadTest2010 databases that are on versions of SQL Server older than 2012. The automatic Interval Update sprocs will not work on earlier versions. Therefore I currently do not automatically call the UpdateLoadTestIntervalTable sproc from the built in sprocs. I will post that in a future blog entry.)

• Prc_UpdateSummaryData3 – Adds the following code at the very end of the stored procedure:

----------------------------------------------------------------
-- Added by the VSLT Reporting Tool
EXEC VSLT_prc_UpdateLoadTestCoreInfoTable
----------------------------------------------------------------

• Prc_DeleteLoadTestRun – Adds the data tables to the list of all tables in the DB (in the proper order). It then adds the calls to delete data from the new tables using the same criteria as the main delete portion.

Warehousing tables and update sprocs.sql

A Note About this Particular Series of Posts

I am starting off fairly simple with the sprocs I am showing here. However, I am leading up to some fairly complex calculations as well as data warehousing. I will be showing you new tables I add to the DB soon that help with a number of calculations, but since they rely on much of the same code as the simple ones, I want to explain these first.

The start of every report

We always need to use the LoadTestRunId to define what run(s) we want to work on. To get that, we use a simple query:

SELECT LoadTestRunId, LoadTestName, Description, StartTime, Outcome FROM LoadTestRun

For the rest of this post, we will use either run #4 or run #52

Page Response Times

The first sproc is VSLT_prc_Page_ResponseTimes. There are also similar sprocs for tests and transactions. There are 5 parameters you can pass in, but only one is required (the LoadTestRunId). If you do not supply any of the others, they will use default values:

@RunId  INT
,@StartOffset INT = NULL
,@EndOffset INT = NULL
,@NameFilter nvarchar(255) = '%'
,@NumberOfRows INT = 25

To show the basic syntax, here is the call and results for an entire run, in this case run #4:

EXEC    VSLT_prc_Page_ResponseTimes
        @RunId = 4

Now, let’s look at the graph of the user load for the run (image below). Here we can see that the load steps up throughout the run [the highlighted red line], so looking at an overall average is misleading. We want to be able to look at the values during each constant load portion. I chose the portion between 2 minutes 10 seconds and 4 minutes.

To get the values there, I add two more of the parameters to the call to the sproc, and it now calculates the values using only the page responses that were received during the time frame specified. NOTE: The offsets are specified using seconds and are based off of time from the start of the run, so 2 minutes 10 seconds = 130

EXEC    VSLT_prc_Page_ResponseTimes
        @RunId = 4,
        @StartOffset = 130,
        @EndOffset = 240

And here are the results [NOTE: The last two columns; Scenario and TestName both use one of the VSLT Views to get the proper names to add here]:

The Way It Works

The heart of the sproc is the WHERE clause filter. The rest of the sproc is all of the joins and the temp tables to allow for multiple queries to be combined (this is required in order to get the 90% and 95% values). Let’s break it down and then you can modify it however you want if you need different results.

WHERE 
a.LoadTestRunId = @RunId

Pick only this run

AND (@StartOffset IS NULL OR c.EndTime >= DATEADD(second, @StartOffset, b.StartTime))

If no start offset was provided, ignore this part of the filter. If it was provided, then filter results to only those which have an end time greater than or equal to the time of the specified offset.

AND (@EndOffset IS NULL OR c.EndTime < DATEADD(second, @EndOffset, b.StartTime))

If no end offset was provided, ignore this part of the filter. If it was provided, then filter results to only those which have an end time less than the time of the specified offset.

AND d.RequestUri LIKE @NameFilter

I cheat here and provide a global value for the name filter, which makes the query slower since it has to scan the entire table, but it simplifies the use of the query. I will probably update this in a future version.

AND (c.Outcome = 0)

This filter is hardcoded to only look at requests whose outcome was PASS.

AND (a.InMeasurementInterval = 1)

This filter is hardcoded to exclude any requests that were captured during warmup or cooldown.

Page Throughput

This sproc is the same as the Page Response Time except that it focuses on calculating the number of pages per second for each page. It takes the total number of pages executed in a specified interval and uses the interval’s duration to get the final value. As with the response time sproc, it takes between 1 and 5 args (the same five). Here is the output for the throughput sproc for the same run as above and the same interval time:

EXEC    VSLT_prc_Page_Throughput
        @RunId = 4,
        @StartOffset = 130,
        @EndOffset = 240

I could have combined the throughput calculation with the response time sproc, but you will see later on when I get into multiple runs and other parts that it is easier to keep these separate. However, since you are getting access to the code, feel free to handle it however you would like.

Grouping By Name

The last part I want to show here is calculating the response times grouped by their name. Sometimes you may have multiple web tests or scenarios that contain calls to the same page. As I explained in an earlier post, Visual Studio tracks each of these separately. However, you can use this sproc to get the response times by using the request URL or reporting name as the grouping mechanism. Take a look at the two tables below. The first shows the response times for a test I ran that had a timer call in two different locations. The second shows the same test results, but with both instances of the call combined and recalculated:

Conclusion

Hopefully you will start to see how you can use the data in the Database to make more informed evaluations of the results. As we delve into the next part, I will setup some tables that can be used to build quick dashboards and also to allow for more automation of much of the interval calculations shown in this post. As always, please provide feedback.

The sql file attached contains the following stored procedures:

• VSLT_prc_Page_ResponseTimes
• VSLT_prc_Page_Throughput
• VSLT_prc_Page_ResponseTimes_GroupedByName
• VSLT_prc_Test_ResponseTimes
• VSLT_prc_Transaction_ResponseTimes
• VSLT_prc_Transaction_ResponseTimes_GroupedByName

VSLT Sprocs Round1.sql

The Mystery of the Perfmon Page Counters

There are two ways you can look at the details for pages, transactions, etc in the results DB. You can look at the detailed timing tables or you can look at the perfmon data (see this post for details on the differences). Each has a distinct advantage. However, getting data from the perfmon counters can be tricky since the database combines two distinct columns into one for the Instance value.

When I customize my reporting, I use a lot of stored procedures that rely heavily on pulling data from perfmon. In order to make this easier, I create a view that exposes the counters in a way that allows me to use the perfmon values the same way I would use the Page Details. Below is the code for creating the view (note, in the attached zip, I have included some other views that will get used in future posts.

CREATE VIEW VSLT_View_PageNamesWithIDs AS
    SELECT a.LoadTestRunId
        , c.ScenarioId
        , ScenarioName
        , a.TestCaseId
        , TestCaseName
        , RequestId
        , RequestUri
        ,RequestUri + '('
            + RIGHT ('000'
            + CAST(RequestId AS nvarchar(3)), 3)
           + ')' AS RequestNameWithId
    FROM dbo.WebLoadTestRequestMap AS a

    INNER JOIN LoadTestCase AS b
        ON a.LoadTestRunId = b.LoadTestRunId
        AND a.TestCaseId = b.TestCaseId

    INNER JOIN LoadTestScenario AS c
        ON b.LoadTestRunId = c.LoadTestRunId
        AND b.ScenarioId = c.ScenarioId
GO

When we use this in a query, we get the proper names and IDs to use elsewhere:

SELECT * FROM VSLT_View_PageNamesWithIDs WHERE LoadTestRunId = 4

In the table above, you can see how you get the proper request ID, which can be used to get info from the LoadTestPageDetails table, and you can determine which test or scenario the page was a part of. The key to this is to build the RequestNameWithId out of the request URI and the request ID. Visual Studio stores the page names and ID as a combined coluimn in the LoadTestPerformanceCounterInstance table. I will show how I incorporate this into creating results in the next post.

Getting Formatted Perfmon Names From The DB

The next view set I use is for populating some core tables and for Perfmon Import/Export tool. This view lays out the data in the three Perfmon naming tables in a way that is ready to add to graphs and tables, and makes adding filters to your queries easier when you are trying to get the Perfmon Instance ID that maps to the actual samples in the LoadTestPerformanceCounterSample table. The view does the typical INNER JOINs of the tables, but it also performs one other critical piece. Visual Studio uses a reserved word to indicate the instance for any counter that does not have instances available.

For example, the perfmon counter Memory/Available MBytes does not define instances. Because of this, the instance name would be NULL and that is not a good thing in this type of schema. So Visual Studio inserts the keyword systemdiagnosticsperfcounterlibsingleinstance in the instance name. In order to make items look good in reports, you need to remove that name, therefore the VIEW has the code to handle it.

Finally, the view also sets the order of the combined name in a format that is compatible with Perfmon CSV naming conventions:

Machine\Category(Instance)\Counter

Here is the code for the view:

CREATE VIEW VSLT_View_PerfmonInstanceNamesAndIds AS
    SELECT
        instance.LoadTestRunId
        ,instance.InstanceId
        ,(
            '\\' + category.MachineName
            + '\' + category.CategoryName
           +    case instance.InstanceName when 'systemdiagnosticsperfcounterlibsingleinstance'
                        then ''
                        else '(' + instance.InstanceName  + ')'
                        end
            + '\' + counter.CounterName
        ) AS InstanceName
        ,category.MachineName
        ,category.CategoryName
        ,counter.CounterName
        ,(case instance.InstanceName when 'systemdiagnosticsperfcounterlibsingleinstance'
                        then ''
                        else instance.InstanceName
                        end) AS Instance
        ,instance.LoadTestItemId

    FROM LoadTestPerformanceCounterCategory AS category

    INNER JOIN LoadTestPerformanceCounter AS counter
        ON category.LoadTestRunId = counter.LoadTestRunId
        AND category.CounterCategoryId = counter.CounterCategoryId

    INNER JOIN LoadTestPerformanceCounterInstance AS instance
        ON counter.CounterId = instance.CounterId
        AND counter.LoadTestRunId = instance.LoadTestRunId
GO

Here is a query that uses the view:

SELECT * FROM VSLT_View_PerfmonInstanceNamesAndIds WHERE LoadTestRunId = 4 AND CategoryName IN ('Memory','Processor')

Conclusion

In the next post, I will show you some stored procedures that take advantage of these views, as well as some of the other views in the attached sql file. The sql file contains the following views:

• VSLT_View_PerfmonInstanceNamesAndIds
• VSLT_View_PageSummary
• VSLT_View_PageNamesWithIDs
• VSLT_View_TestNamesWithIDs
• VSLT_View_TransactionNamesWithIDs

To use it, simply open it in a SQL editor (Visual Studio or SSMS) and execute it against the LoadTest2010 database.

VSLT_Views.sql

The Excel Reporting Feature

In the 2010 release of Visual Studio Ultimate, a new feature was added that allows you to use Microsoft Excel to create either a Run Comparison or Trend Analysis report. You can read the standard documentation to learn more about how to use the feature.

The feature uses Office COM Interop to interact with Excel by walking through a wizard, asking questions about how to report, letting you choose which perfmon counters to display, then creating different worksheets for each category chosen.

Caveats

The reporting add-in has a few noteable limitations:

• You can only select runs that were all created with the same loadtest name.
• Once a report is created, you cannot change the inputs to it. You will need to create a new instance of the report to modify it (NOTE: You can use the existing report as a template to make creating the new report faster).
• The templates that get created save the data categories to collect, but do NOT save any EXCEL formatting.
• You cannot choose the time range(s) for the reports.
• You must have access to Excel 2010 or newer, and it must be installed on the computer where you are viewing the test results.

Examples

The next section shows an example of the SQL calls made from Excel for each of the report types. I selected a bare minimum of counters to use for each report.

Building a Run Comparison Report

When you select the “Open and Manage Results” dialog box, Visual Studio queries the LoadTestRun table and retrieves a list of available runs to choose from:

Query: SELECT Name FROM sys.Databases ORDER BY 1 ASC

Sproc: exec prc_QueryLoadTestReports

Sproc: exec prc_FindLoadTestReport @reportId=-1,@name=N'LOADTEST_RUNCOMPARISON_REPORT_DEFAULT'

Query: SELECT DISTINCT LoadTestName FROM LoadTestRun ORDER BY 1 ASC

Sproc: exec sp_executesql N'SELECT COUNT(*) FROM LoadTestReport WHERE Name = @ReportName',N'@ReportName nvarchar(16)',@ReportName=N'Some Report Name'

Sproc: exec prc_QueryLoadTestRuns @loadTestName=N'LoadTest1'

Sproc: exec prc_QueryPossibleCountersForReport @runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_CreateLoadTestReport @name=N'Some Report Name',@reportType=2,@description=N'',@loadTestName=N'LoadTest1', @lastModifiedBy=N'NORTHAMERICA\geoffgr',@lastRunId=2172, @selectNewReports=1,@runsXml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2158" /><r i="2157" /></runs>',@pageXml=N'<?xml version="1.0" encoding="utf-16"?><pages><p i="0" c="LoadTest:Page" t="Avg. Page Time" />...[truncated]...</pages>'

Sproc: exec Prc_QueryLoadTestTestComparison @Baseline=2157,@ComparisonRun=2158

Sproc: exec Prc_QueryLoadTestPageComparison @Baseline=2157,@ComparisonRun=2158

Sproc: exec Prc_QueryLoadTestTransactionComparison @Baseline=2157,@ComparisonRun=2158

Sproc: exec prc_QueryForMachinesInRun @Baseline=2157,@ComparisonRun=2158

Sproc: exec Prc_QueryComputedCounterComparison @ReportId=4,@Baseline=2157,@ComparisonRun=2158

Sproc: exec Prc_QueryLoadTestErrorComparison @Baseline=2157,@ComparisonRun=2158

Building a Trend Report

Once you select a run and open it, there are a number of queries and Stored Procs that get called, filling DataSets in Visual Studio and allowing you to look at different graphs and tables. The bulk of core data is grabbed immediately and there are only a few instances where more trips to SQL will occur. Below is a list SQL calls made by selecting to open a single run. A few of the longer queries were truncated to save room on this post. You can run a SQL Profiler trace and capture all of the text yourself if you need to dig in deeper.:

Query: SELECT Name FROM sys.Databases ORDER BY 1 ASC

Sproc: exec prc_QueryLoadTestReports

Sproc: exec prc_FindLoadTestReport @reportId=-1,@name=N'LOADTEST_TREND_REPORT_DEFAULT'

Query: SELECT DISTINCT LoadTestName FROM LoadTestRun ORDER BY 1 ASC

Sproc: exec sp_executesql N'SELECT COUNT(*) FROM LoadTestReport WHERE Name = @ReportName',N'@ReportName nvarchar(5)',@ReportName=N'Trend'

Sproc: exec prc_QueryLoadTestRuns @loadTestName=N'00_LoadTest1'

Sproc: exec sp_executesql N'SELECT COUNT(*) FROM LoadTestReport WHERE Name = @ReportName',N'@ReportName nvarchar(5)',@ReportName=N'Trend'

Sproc: exec prc_QueryLoadTestRuns @loadTestName=N'LoadTest1'

Sproc: exec prc_QueryPossibleCountersForReport @runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_CreateLoadTestReport @name=N'Trend',@reportType=1,@description=N'',@loadTestName=N'LoadTest1', @lastModifiedBy=N'NORTHAMERICA\geoffgr',@lastRunId=2172, @selectNewReports=1,@runsXml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>',@pageXml=N'<?xml version="1.0" encoding="utf-16"?><pages><p i="0" c="LoadTest:Page" t="Avg. Page Time" /><p i="0" c="LoadTest:Page" t="Pages/Sec" />...[truncated]...</pages>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Page',@counterName=N'Avg. Page Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Page',@counterName=N'Pages/Sec',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Request',@counterName=N'Requests/Sec', @runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Scenario',@counterName=N'User Load',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Test',@counterName=N'Avg. Test Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Test',@counterName=N'Total Tests',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Transaction',@counterName=N'Avg. Response Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Transaction',@counterName=N'Total Transactions',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'LoadTest:Transaction',@counterName=N'Transactions/Sec', @runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'Memory',@counterName=N'Available MBytes',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'Network Interface',@counterName=N'Bytes Total/sec',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'PhysicalDisk',@counterName=N'% Idle Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'Process',@counterName=N'% Processor Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: exec prc_QueryForInstanceCount @categoryName=N'Processor',@counterName=N'% Processor Time',@runsxml=N'<?xml version="1.0" encoding="utf-16"?><runs><r i="2160" /><r i="2159" /><r i="2158" /><r i="2157" /></runs>'

Sproc: Prc_QueryLoadTestPageSummary "5" , "Avg. Page Time"

Sproc: Prc_QueryLoadTestPageSummary "5" , "Avg. Page Time"

Sproc: Prc_QueryLoadTestPageSummary "5" , "Pages/Sec"

Sproc: Prc_QueryLoadTestPageSummary "5" , "Pages/Sec"

Sproc: Prc_QueryLoadTestRequestSummary "5" , "Requests/Sec"

Sproc: Prc_QueryLoadTestRequestSummary "5" , "Requests/Sec"

Sproc: Prc_QueryLoadTestScenarioSummary "5" , "User Load"

Sproc: Prc_QueryLoadTestScenarioSummary "5" , "User Load"

Sproc: Prc_QueryLoadTestTestCaseSummary "5" , "Avg. Test Time"

Sproc: Prc_QueryLoadTestTestCaseSummary "5" , "Avg. Test Time"

Sproc: Prc_QueryLoadTestTestCaseSummary "5" , "Total Tests"

Sproc: Prc_QueryLoadTestTestCaseSummary "5" , "Total Tests"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Avg. Response Time"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Avg. Response Time"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Total Transactions"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Total Transactions"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Transactions/Sec"

Sproc: Prc_QueryLoadTestTransactionSummary "5" , "Transactions/Sec"

Sproc: Prc_QueryComputedCounterSummary "5" , "Available MBytes" , "Memory"

Sproc: Prc_QueryComputedCounterSummary "5" , "Available MBytes" , "Memory"

Sproc: Prc_QueryComputedCounterSummary "5" , "Bytes Total/sec" , "Network Interface"

Sproc: Prc_QueryComputedCounterSummary "5" , "Bytes Total/sec" , "Network Interface"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Idle Time" , "PhysicalDisk"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Idle Time" , "PhysicalDisk"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Processor Time" , "Process"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Processor Time" , "Process"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Processor Time" , "Processor"

Sproc: Prc_QueryComputedCounterSummary "5" , "% Processor Time" , "Processor"

Next: “Scalpel Please”  

In the next post, we will start our deep dive into “Slicing and Dicing” the data to show you what really happened.

The High Level Schema

The main schema for the database was created for the first version of VSTS in 2005. Since then the schema has been extended twice, but not changed. This is important because it means that any results from load tests run in VSTS 2005 are still valid and can still be viewed inside Visual Studio.

Every table in the database has a column for LoadTestRunId. This Id is generated for every single load test execution that is stored in the database. Before Visual Studio starts the execution of a test, it adds a row to the LoadTestRun table, which triggers the creation of a unique integer via SQL’s Identity property. Then it grabs that value and uses it throughout the rest of the execution and the writing of the results. (NOTE: You can use this step to query the results database inside a LoadTest plugin and capture the Id so you can add your own custom items to the database during a run. Bill Barnett helped me back in 2008 to do this with console messages for a SQL CE simulation load test harness using unit tests. The LoadTestRun table is the primary “list” for all runs available. There is one row for each load test that has been executed (or is in the process of being executed).

The rest of the data is heavily normalized and most tables use a combination of the LoadTestRunId and other columns to create multi-column primary keys. The data tables are broken down into a few categories below, and the views (listed separately) are used to “un-normalize the data for reporting purposes. The stored procedures will be covered in the next article.

The Built In Tables

• Core Run Tables
  • LoadTestRun: Primary Run Table - One row per load test run
  • LoadTestSchemaRevision: For VS 2010 - 2015, this will have a single entry with the value of 3
  • LoadTestCase: One row for each inner test referenced in the .loadtest file
  • LoadTestFileAttachment: Used to store info about attachments from things like data collectors, etc
  • LoadTestFileAttachmentChunk: Used to store the actual attachments from the above table
  • LoadTestDataCollectorLog: Used to store info about the data collectors used during the run
  • LoadTestTestLog: Stores the full testcase results of a test iteration that is logged
  • LoadTestSqlTrace: Stores SQL Proifile data from tests where SQL Tracing is enabled

• Core Detailed Timings Tables (These tables may not contain data for some runs. See this postfor more information)
  • LoadTestTestDetail: Stores timing and outcome information for every iteration of every test executed.
  • LoadTestPageDetail: Stores timing and outcome information for every iteration of every primary level request executed.
  • LoadTestTransactionDetail: Stores timing information for every iteration of every transaction that successfully completed. If a request fails while inside a transaction, the transaction will not have an entry.
  • LoadTestDetailMessage: One row for every message that occurs during a test run. This is the master list of errors.

• Core Timings Summary Tables (Qty, Min, Avg, Max, 90%, 95%, 99%, Mean, Std Dev)  
  • LoadTestTestSummaryData: Stores the summary of the LoadTestTestDetail data
  • LoadTestTransactionSummaryData: Stores the summary of the LoadTestTransactionDetail data
  • LoadTestPageSummaryData: Stores the summary of the LoadTestPageDetail data
  • LoadTestPageSummaryByNetwork: Stores the summary of the LoadTestPageDetail data, broken out by network emulator (if applicable)

• Tables to map names to IDs
  • WebLoadTestErrorDetail: (normalized data) Maps error messages to the web pages that caused the errors
  • WebLoadTestRequestMap: (normalized data) Maps the request/page name to the PageId for the above tables
  • WebLoadTestTransaction: (normalized data) Maps the transaction name to the transactionID for the above tables

• Error Message Tables
  • LoadTestMessage: The details for any message (error, etc) captured during execution
  • LoadTestMessageType: (normalized data) the type of message for the above table
  • LoadTestThresholdMessage: The details for any threshold violation captured during execution

• Perfmon Tables (NOTE: The way the data is stored for the pseudo counters [e.g. Test, Transaction, Page, etc] will be discussed in a separate post as part of this series.)
  • LoadTestPerformanceCounterCategory: (normalized data) Stores the perfmon CATEGORY and MACHINE for each perfmon category collected (One row per Machine/Category pair)
  • LoadTestPerformanceCounter: (normalized data) Stores one row for each COUNTER collected inside each LoadTestPerformanceCounterCategory entry
  • LoadTestPerformanceCounterInstance: (normalized data) Stores one row for each counter INSTANCE collected inside each LoadTestPerformanceCounter entry
  • LoadTestPerformanceCounterSample: The actual perfom sampled values. Contains one row for each item in the LoadTestPerformanceCounterInstance table for each LoadTestRunInterval

• SUT and RIG settings tables
  • LoadTestRunAgent: (normalized data) One entry for each agent that was used during test execution
  • LoadTestRunInterval: (normalized data) One entry for each sample interval during the test
  • LoadTestScenario: (normalized data) One entry for each Scenario defined in the .loadtest file
  • LoadTestBrowsers: (normalized data) One entry for each browser type emulated in the test
  • LoadTestNetworks: (normalized data) One entry for each network type emulated in the test
  • LoadTestSystemUnderTest: (normalized data) One entry for each machine that had perfmon data collected and that was NOT a part of the test rig
  • LoadTestSystemUnderTestTag: (normalized data) One row for each machine in LoadTestSystemUnderTest that contains a custom identifier tag

• Excel Reporting historical data tables
  • LoadTestReport: Stores configuration info about any Excel reports that have been created tyhrough the Visual Studio IDE
  • LoadTestReportRuns: (normalized data)
  • LoadTestReportPage: (normalized data)

The Built In Views

• Summary Views
  • LoadTestMessageSummary: Combines the LoadTestMessage, LoadTestCase, LoadTestScenario, LoadTestRunAgent, WebLoadTestRequestMap and WebLoadTestErrorDetail tables to provide a summary of the errors and threshold messages (Defines: TestError, Exception, HttpError, ValidationRuleError, ExtractionRuleError, Timeout, DataCollectionError, DataCollectionWarning, Unknown).
  • LoadTestMessageView: Combines the LoadTestMessage, LoadTestCase, LoadTestScenario, LoadTestRunAgent, WebLoadTestRequestMap and WebLoadTestErrorDetail tables to provide details about the errors and threshold messages.
  • LoadTestMessageView2: Combines the LoadTestMessage, LoadTestCase, LoadTestScenario, LoadTestRunAgent, WebLoadTestRequestMap, WebLoadTestErrorDetail and LoadTestTestLog tables to provide details about the errors and threshold messages.
  • LoadTestPageResults: Combines the LoadTestPageSummaryData, WebLoadTestRequestMap, LoadTestCase and LoadTestScenario tables to provide a summary of page performance based on actaul page data.
  • LoadTestPageResultsByNetwork: Combines the LoadTestPageSummaryByNetwork, WebLoadTestRequestMap, LoadTestCase, LoadTestScenario and LoadTestNetworks tables to provide a summary of page performance split into the different networks emulated based on actaul page data.
  • LoadTestPageResultsByNetwork2: Provides the same info as LoadTestPageResultsByNetwork, except it adds 99%, Std Dev and Mean
  • LoadTestTestResults: Combines the LoadTestTestSummaryData, LoadTestCase and LoadTestScenario tables to provide a summary of test performance based on actaul test data.
  • LoadTestTransactionResults: Combines the LoadTestTransactionSummaryData, WebLoadTestTransaction, LoadTestCase and LoadTestScenario tables to provide a summary of transaction performance based on actaul transaction data.
  • LoadTestTransactionResults2: Provides the same info as LoadTestTransactionSummaryData, except it adds 99%, Std Dev and Mean

• Perfmon Data Views
  • LoadTestComputedCounterSample: Combines the LoadTestPerformanceCounterCategory, LoadTestPerformanceCounter, LoadTestPerformanceCounterInstance, LoadTestPerformanceCounterSample and LoadTestRunInterval tables to provide perfmon sample values.
  • LoadTestComputedCounterSummary: Combines the LoadTestRun, LoadTestPerformanceCounterCategory, LoadTestPerformanceCounter and LoadTestPerformanceCounterInstance tables to provide perfmon cumulative (summary) values.
  • LoadTestPageSample: Combines the necessary tables to provide a sampled data of page performance based on sampled Perfmon Counters (Used for graphing results).
  • LoadTestPageSummary: Combines the necessary tables to provide a summary of page performance based on sampled Perfmon Counters.
  • LoadTestRequestSummary: Combines the necessary tables to provide a summary of request performance based on sampled Perfmon Counters.
  • LoadTestScenarioSummary: Combines the necessary tables to provide a summary of scenario performance based on sampled Perfmon Counters.
  • LoadTestTestCaseSample: Combines the necessary tables to provide a sampled data of test case performance based on sampled Perfmon Counters (Used for graphing results).
  • LoadTestTestCaseSummary: Combines the necessary tables to provide a summary of test case performance based on sampled Perfmon Counters.
  • LoadTestTransactionSample: Combines the necessary tables to provide a sampled data of transaction performance based on sampled Perfmon Counters (Used for graphing results).
  • LoadTestTransactionSummary: Combines the necessary tables to provide a summary of transaction performance based on sampled Perfmon Counters.
  • LoadTestWebPageAndUserLoadCounterSample: Combines the necessary tables to provide a summary of request Throughput based on sampled Perfmon Counters.
  • LoadTestWebRequestCounterSample: Combines the necessary tables to provide a sampled data of request performance based on sampled Perfmon Counters (Used for graphing results).
  • LoadTestWebRequestCounterSummary: Combines the necessary tables to provide a summary of total vs. passed request counts.

The Sprocs

The following list shows all of the built in stored procedures, broken down by the type of functionality they provide. [NOTE: All sprocs that have a 2 or 3 at the end denote a schema revision change where functionality was ADDED to the results. For any of these sprocs, the Schema Revision number stored with each run is used to determine which sproc to call. If the stored schema revision is higher than any of the sproc versions, the highest available sproc version is used.]:

• Create/Delete- These are used to control creation and modification of data
  • Prc_DeleteDataCollectorLogs
  • Prc_DeleteLoadTestRun
  • prc_FindLoadTestReport
  • prc_GrantDBAccess
• Updating Results - These are called just after all results are written to the database at the end of a run. They perform the updates to the summary tables. See this article for more information.
  • Prc_UpdateInstanceSummaryData
  • prc_UpdateLoadTestReport
  • Prc_UpdatePageByNetworkPercentiles
  • Prc_UpdatePagePercentiles
  • Prc_UpdatePageSummary
  • Prc_UpdatePageSummary3
  • Prc_UpdatePageSummaryByNetwork
  • Prc_UpdatePageSummaryByNetwork3
  • Prc_UpdateSummaryData
  • Prc_UpdateSummaryData2
  • Prc_UpdateSummaryData3
  • Prc_UpdateTestPercentiles
  • Prc_UpdateTestSummary
  • Prc_UpdateTestSummary3
  • Prc_UpdateTransactionPercentiles
  • Prc_UpdateTransactionSummary
  • Prc_UpdateTransactionSummary3
• Get Misc Info- These [along with various direct queries] grab various pieces of information for generating the results shown inside the Visual Studio IDE
  • Prc_GetAgents
  • Prc_GetAvailableDetailTypes
  • Prc_GetCategories
  • Prc_GetCompletedLoadTestRuns
  • Prc_GetErrorDetail
  • Prc_GetFirstSampleForInstance
  • Prc_GetInitialDetailRange
  • Prc_GetInstanceSummaries
  • Prc_GetLastSampleForInstance
  • Prc_GetRequestMap
  • Prc_GetRequestMap2
  • Prc_GetSamplesForInstance
  • Prc_GetTestCases
  • Prc_GetTestLog
  • Prc_GetTransactionMap
  • Prc_GetUniqueDetailIds
  • Prc_GetUserPageDetail
  • Prc_GetUserTestDetail
  • Prc_GetUserTransactionDetail
• Error Info- These get information about any errors that occurred during the run, including the details and the actual results, if available.
  • Prc_GetMessagesAll
  • Prc_GetMessagesAll2
  • Prc_GetMessagesByAgent
  • Prc_GetMessagesByAgent2
  • Prc_GetMessagesByRequest
  • Prc_GetMessagesByRequest2
  • Prc_GetMessagesByTest
  • Prc_GetMessagesByTest2
  • Prc_GetMessagesByType
  • Prc_GetMessagesByType2
  • Prc_GetMessagesTypeSummary
• Insertion Routines- These procs are called as part of the test execution [and 'Writing Results'] in order to store the data
  • Prc_InsertFileAttachmentChunk
  • Prc_InsertLoadTestDetailMessage
  • Prc_InsertLoadTestMessageType
  • Prc_InsertPageDetail
  • Prc_InsertPageDetail2
  • Prc_InsertSample
  • Prc_InsertSqlTraceRow
  • Prc_InsertTestDetail
  • Prc_InsertTestDetail2
  • Prc_InsertTestLog
  • Prc_InsertTransactionDetail
  • Prc_InsertTransactionDetail2
• Procs for Excel Reporting- These are used by Excel as part of the Excel Reporting AddIn provided by Visual Studio.
  • prc_CreateLoadTestReport
  • Prc_QueryComputedCounterComparison
  • Prc_QueryComputedCounterSummary
  • prc_QueryForInstanceCount
  • prc_QueryForMachinesInRun
  • Prc_QueryLoadTestErrorComparison
  • Prc_QueryLoadTestPageComparison
  • Prc_QueryLoadTestPageSummary
  • prc_QueryLoadTestReports
  • Prc_QueryLoadTestRequestSummary
  • prc_QueryLoadTestRuns
  • prc_QueryLoadTestRunsById
  • Prc_QueryLoadTestScenarioSummary
  • Prc_QueryLoadTestTestCaseSummary
  • Prc_QueryLoadTestTestComparison
  • Prc_QueryLoadTestTransactionComparison
  • Prc_QueryLoadTestTransactionSummary
  • prc_QueryPossibleCountersForReport

Examples

The next section shows an example of the work that Visual Studio performs when opening a result from the database. The information was captured from a run that had been exported from a full test rig and loaded into my Surface Pro 2. I had not added any extra graph objects to the results and all graphing options are still set to default. When you open runs, the queries, sprocs and arguments passed into both will vary. The purpose of this is to allow you to see a fairly representative sample of the data flow between Visual Studio and SQL.

Opening the Manage Results Dialog

When you select the “Open and Manage Results” dialog box, Visual Studio queries the LoadTestRun table and retrieves a list of available runs to choose from:

select LoadTestRunId, RunId, LoadTestName, Description, Comment, IsLocalRun, ControllerName, StartTime, EndTime, WarmupTime, RunDuration, LoadTest, Outcome, LoadTestSchemaRev,CooldownTime from LoadTestRun where LoadTestRunId = 2156

Opening a Result

Once you select a run and open it, there are a number of queries and Stored Procs that get called, filling DataSets in Visual Studio and allowing you to look at different graphs and tables. The bulk of core data is grabbed immediately and there are only a few instances where more trips to SQL will occur. Below is a list SQL calls made by selecting to open a single run. A few of the longer queries were truncated to save room on this post. You can run a SQL Profiler trace and capture all of the text yourself if you need to dig in deeper.:

Query: select LoadTestRunId, RunId, LoadTestName, Description, Comment, IsLocalRun, ControllerName, StartTime, EndTime, WarmupTime, RunDuration, LoadTest, Outcome, LoadTestSchemaRev,CooldownTime from LoadTestRun where LoadTestRunId = 2156

Sproc: exec Prc_GetRequestMap2 @LoadTestRunId=2156

Sproc: exec Prc_GetCategories @LoadTestRunId=2156

Query: SELECT TestCaseId,Percentile90,Percentile95 FROM LoadTestTestSummaryData WHERE LoadTestRunId = 2156

Query: SELECT TransactionId,Percentile90,Percentile95 FROM LoadTestTransactionSummaryData WHERE LoadTestRunId = 2156

Query: SELECT PageId,Percentile90,Percentile95 FROM LoadTestPageSummaryData WHERE LoadTestRunId = 2156

Sproc: exec Prc_GetInstanceSummaries @LoadTestRunId=2156

Query: SELECT TOP 1000 category.MachineName, category.CategoryName, [truncated]

Query: exec Prc_GetMessagesTypeSummary @LoadTestRunId=2156

Query: select FileAttachmentId, MachineName, Filename, Filesize from LoadTestFileAttachment where LoadTestRunId = 2156

Query: select DataCollectorLogId, DataCollectorDisplayName, MachineName from LoadTestDataCollectorLog where LoadTestRunId = 2156

Query: SELECT ScenarioName, TestCaseName, TransactionName, TransactionCount, Minimum, Maximum, Average, Median, Percentile90, Percentile95, Percentile99, StandardDeviation, AvgTransactionTime FROM LoadTestTransactionResults2 where LoadTestRunId = 2156 ORDER BY ScenarioName, TestCaseName, TransactionName

Query: SELECT ScenarioName, TestCaseName, RequestUri, NetworkName, PageCount, Minimum, Maximum, Average, Median, Percentile90, Percentile95, Percentile99, StandardDeviation, Goal, PagesMeetingGoal FROM LoadTestPageResultsByNetwork2 where LoadTestRunId = 2156 ORDER BY ScenarioName, TestCaseName, RequestUri, NetworkName

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT TOP 1000 Event = CASE EventClass WHEN 10 THEN 'RPCCompleted'

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Query: SELECT samples.InstanceId, interval.IntervalEndTime, samples.RawValue, [truncated]

Adding a Graph Item

Many of the graph items that you might add will already be stored in the DataSets in Visual Studio. However, if you choose to add a perfmon counter to a graph that has not been loaded, the following will get executed:

Query: SELECT samples.InstanceId,interval.IntervalEndTime, samples.RawValue, samples.BaseValue, samples.CounterFrequency, samples.SystemFrequency, samples.SampleTimeStamp, samples.SampleTimeStamp100nSec, samples.CounterType, samples.ThresholdRuleResult FROM LoadTestPerformanceCounterSample as samples LEFT OUTER JOIN LoadTestRunInterval AS interval ON samples.LoadTestRunId = interval.LoadTestRunId AND samples.TestRunIntervalId = interval.TestRunIntervalId WHERE samples.LoadTestRunId = 2156 AND (samples.InstanceId = 93) ORDER BY samples.InstanceId, samples.TestRunIntervalId

Viewing a Table Item

Table items such as Tests, Transactions, Pages, etc. are already in the DataSets, but a couple of tables (such as Threshold and SQL Trace) are not pre-loaded. The following shows what will execute if you select the Threshold table:

Query: SELECT TOP 1000 category.MachineName, category.CategoryName, counter.CounterName, instance.InstanceName, interval.IntervalEndTime, countersample.ThresholdRuleResult, thresholdMessage.MessageText FROM LoadTestPerformanceCounterCategory AS category INNER JOIN LoadTestPerformanceCounter AS counter ON category.LoadTestRunId = counter.LoadTestRunId AND category.CounterCategoryId = counter.CounterCategoryId INNER JOIN LoadTestPerformanceCounterInstance AS instance ON counter.CounterId = instance.CounterId AND counter.LoadTestRunId = instance.LoadTestRunId INNER JOIN LoadTestPerformanceCounterSample AS countersample ON countersample.InstanceId = instance.InstanceId AND countersample.LoadTestRunId = instance.LoadTestRunId INNER JOIN LoadTestRunInterval AS interval ON interval.LoadTestRunId = countersample.LoadTestRunId AND interval.TestRunIntervalId = countersample.TestRunIntervalId INNER JOIN LoadTestThresholdMessage AS thresholdMessage ON interval.LoadTestRunId = thresholdMessage.LoadTestRunId AND interval.TestRunIntervalId = thresholdMessage.TestRunIntervalId AND countersample.ThresholdRuleMessageId = thresholdMessage.MessageId WHERE countersample.ThresholdRuleMessageId IS NOT NULL AND thresholdMessage.LoadTestRunId = 2156 ORDER BY interval.IntervalEndTime ASC

The Details Tab

The following sprocs and queries get executed whenever you switch to the “Details” tab:

Sproc: exec Prc_GetUniqueDetailIds @LoadTestRunId=2156,@DetailType=0

Query: select MessageTypeId, MessageType, SubType from LoadTestMessageType where LoadTestRunId = 2156

Sproc: exec Prc_GetAvailableDetailTypes @LoadTestRunId=2156

Next: Excel…

In the next post, we will dive into the sprocs and queries used for the built in Excel Reporting.

Each model can easily be selected regardless of what webtests you have created and both can easily be ramped up to different load levels. The primary difference between the two is how Visual Studio handles the think time between each iteration of the virtual users:

• “Based on Number Of Tests Started” executes the test and then waits for the amount of time provided in the “Think Time Between Test Iterations” property for the scenario. The start time for the first iteration for each vUser is the start time of the vUser.
• “Based on User Pace” gets a rough estimate of how long each iteration can execute and still leave enough time for all subsequent iterations to complete. (e.g. 30 tests/user/hour = one test per user every 2 minutes.) The actual start time for the test iteration will be random and can happen anywhere within the calculated window.

Based on User Pace is much more predictable regardless of response times and Based on Number Of Tests Started is much better at driving load as fast as the system can respond. The images below show the user responses for two virtual users in each of four different load tests:

• 100 users – Based on Number Of Tests Started
• 100 users – Based on User Pace
• 1000 users – Based on Number Of Tests Started
• 1000 users – Based on User Pace

The four graphs below are representative of the User Activity graphs in Visual Studio, but simplified to make the explanation easier. The blue boxes represent the execution of a single iteration of a test by a single user. The space in between blue boxes is the Think Time Between Test Iterations. The gray line at the bottom represents the number of minutes into the overall load test execution.

In tests 1 and 2, we see that the response times are the same, but that test 1 completed more iterations (10) than test 2 (5).

In tests 3 and 4, the load against the system has caused the response times to slow down. Test 3 (which is identical to test 1 except for the number of virtual users) now only completes six iterations per user. (NOTE: The iteration in yellow is a test that started but did not complete in the allotted time and therefore isnot counted in the results. See this post for details) Test 4 (which is identical to test 2) still completes 5 iterations per user.

The best way to think about this is to observe the “white” space in the images above. This represents the “think time between iterations.”

• With both load profile models, the System Under Test (SUT) controls the time it takes to complete an iteration.
• With Based on Number Of Tests Started you control the amount of think time (or pacing).
• With Based on User Pace, Visual Studio controls this, and it uses the SUT’s response time to set it.
• If you want to drive load against the system quickly and irrespective of the throughput, then use Based on Number Of Tests Started.
• If you want to drive load against the system and know that you are providing a known throughput, then use Based on User Pace.

Timing Relationships

I recorded a webtest against Wikipedia and created a couple of transaction timers in the test. Then I ran the test in order to get the various timers as Visual Studio collects them. Below you can see that

1. each request gets its own timer regardless of whether it is a primary request, a dependent request or a redirect request.
2. each request that has dependent and/or redirect requests also gets a timer (called a page timer)
3. each transaction timer I added to the test also gets a timer that includes the sum of all page timers for requests inside the transaction.

The Effect of “Stop On Error” with Transaction Timers

Visual Studio webtests have a property called “Stop On Error” which, when set to True will cause the current iteration of the webtest to stop execution as soon as a request fails or the test encounters an error. This is NOT the default behavior, but I recommend that you set it to True because if something fails, chances are that the next requests will probably also fail. I suggest always setting this value to True.

Setting the value:

If the test passes, there is no effect.

If Stop On Error is TRUE and the test fails, the Transaction Timer will NOT be collected or reported.

If Stop On Error is FALSE and the test fails, the Transaction Timer WILL be collected and reported

Avg Transaction Time vs. Avg Response Time

In a load test, you will see two different average values available for transaction timers: (Avg. Transaction Time and Avg. Response Time). The difference is that the Transaction time includes any think times associated with the requests inside the transaction and Response time does NOT include any think time. The below test has two transactions and only SearchResults has a request with Think Time. (NOTE: On the Summary page, the Transaction Time is referred to as Elapsed Time)