Edit

Identify performance regressions by comparing test runs in Azure Load Testing

  • Article

In this article, you'll learn how you can identify performance regressions by comparing test runs in the Azure Load Testing dashboard. The dashboard overlays the client-side and server-side metric graphs for each run, which allows you to quickly analyze performance issues.

You can compare load test runs for the following scenarios:

  • Identify performance regressions between application builds or configurations. You could run a load test at each development sprint to ensure that the previous sprint didn't introduce performance issues.
  • Identify which application component is responsible for a performance problem (root cause analysis). For example, an application redesign might result in slower application response times. Comparing load test runs might reveal that the root cause was a lack of database resources.

Prerequisites

  • An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.

  • An Azure Load Testing resource with a test plan that has multiple test runs. To create a Load Testing resource, see Create and run a load test.

Select test runs

To compare test runs in Azure Load Testing, you'll first have to select up to five runs within a load test. You can only compare runs that belong to the same load test.

A test run needs to be in the Done, Stopped, or Failed state to compare it.

Use the following steps to select the test runs:

  1. Sign in to the Azure portal by using the credentials for your Azure subscription.

  2. Go to your Azure Load Testing resource and then, on the left pane, select Tests.

    Screenshot that shows the list of tests for a Load Testing resource.

    You can also use the filters to find your load test.

  3. Select the test whose runs you want to compare by selecting its name.

  4. Select two or more test runs by selecting the corresponding checkboxes in the list.

    Screenshot that shows a list of test runs and the 'Compare' button.

    You can choose a maximum of five test runs to compare.

Compare multiple test runs

After you've selected the test runs you want to compare, you can visually compare the client-side and server-side metrics for each test run in the load test dashboard.

  1. Select the Compare button to open the load test dashboard.

    Each test run is shown as an overlay in the different graphs.

    Screenshot of the 'Compare' page, displaying a comparison of two test runs.

  2. Optionally, use the filters to customize the graphs.

    Screenshot of the client-side filter controls on the load test dashboard.

    Tip

    The time filter is based on the duration of the tests. A value of zero indicates the start of the test, and the maximum value marks the duration of the longest test run.

Identify performance regressions

You can compare multiple test runs to identify performance regressions. For example, before deploying a new application version in production, you can verify that the performance hasn't degraded.

Use the client-side metrics, such as requests per second or response time, on the load test dashboard to quickly spot performance changes between different load test runs:

  1. Hover over the client-side metrics graphs to compare the values across the different test runs.

    In the following screenshot, you notice that the metric values for Requests per second and Response Time are significantly different. This difference indicates that the application performance dropped significantly between the two test runs.

    Screenshot of the client-side metrics, highlighting the difference in requests per second and response time.

  2. Optionally, use the Requests filter to compare a specific application request in the JMeter script.

    Screenshot of the client-side 'requests' filter, which allows you to filter specific application requests.

Identify the root cause

When there's a performance issue, you can use the server-side metrics to analyze what the root cause of the problem is. Azure Load Testing can capture server-side resource metrics for Azure-hosted applications.

  1. Hover over the server-side metrics graphs to compare the values across the different test runs.

    In the following screenshot, you notice from the Response time and Requests that the application performance has degraded. You can also see that for one test run, the database RU Consumption peeks at 100%. This indicates that the root cause is likely the database Provisioned Throughput.

    Screenshot of the server-side metrics, highlighting the difference in database resource consumption and provisioning throughput.

  2. Optionally, select Configure metrics to add or remove server-side metrics.

    You can add more server-side metrics for the selected Azure app components to further investigate performance problems. The dashboard immediately shows the additional metrics data, and you don't have to rerun the load test.

  3. Optionally, use the Resource filter to hide or show all metric graphs for an Azure component.

Next steps