Define fail criteria for load tests by using Azure Load Testing Preview

In this article, you'll learn how to define test fail criteria for your load tests with Azure Load Testing Preview. Fail criteria let you define performance and quality expectations for your application under load. Azure Load Testing supports various client metrics for defining fail criteria. Criteria can apply to the entire load test, or to an individual request in the JMeter script.

Important

Azure Load Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

  • An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
  • An Azure load testing resource. If you need to create an Azure Load Testing resource, see the quickstart Create and run a load test.

Load test fail criteria

Load test fail criteria are conditions for client-side metrics, that your test should meet. You define test criteria at the load test level in Azure Load Testing. A load test can have one or more test criteria. When at least one of the test criteria evaluates to true, the load test gets the failed status.

You can define test criteria at two levels. A load test can combine criteria at the different levels.

  • At the load test level. For example, to ensure that the total error percentage doesn't exceed a threshold.
  • At the JMeter request level (JMeter sampler). For example, you could specify a threshold for the response time of the getProducts request, but disregard the response time of the sign in request.

You can define a maximum of 10 test criteria for a load test. If there are multiple criteria for the same client metric, the criterion with the lowest threshold value is used.

Fail criteria structure

The format of fail criteria in Azure Load Testing follows that of a conditional statement for a supported metric. For example, ensure that the average number of requests per second is greater than 500.

Fail criteria have the following structure:

  • Test criteria at the load test level: Aggregate_function (client_metric) condition threshold.
  • Test criteria applied to specific JMeter requests: Request: Aggregate_function (client_metric) condition threshold.

The following table describes the different components:

Parameter Description
Client metric Required. The client metric on which the condition should be applied.
Aggregate function Required. The aggregate function to be applied on the client metric.
Condition Required. The comparison operator, such as greater than, or less than.
Threshold Required. The numeric value to compare with the client metric.
Request Optional. Name of the sampler in the JMeter script to which the criterion applies. If you don't specify a request name, the criterion applies to the aggregate of all the requests in the script.

Supported client metrics for fail criteria

Azure Load Testing supports the following client metrics:

Metric Aggregate function Threshold Condition Description
response_time_ms avg (average)
min (minimum)
max (maximum)
pxx (percentile), xx can be 50, 90, 95, 99
Integer value, representing number of milliseconds (ms). > (greater than)
< (less than)
Response time or elapsed time, in milliseconds. Learn more about elapsed time in the Apache JMeter documentation.
latency_ms avg (average)
min (minimum)
max (maximum)
pxx (percentile), xx can be 50, 90, 95, 99
Integer value, representing number of milliseconds (ms). > (greater than)
< (less than)
Latency, in milliseconds. Learn more about latency in the Apache JMeter documentation.
error percentage Numerical value in the range 0-100, representing a percentage. > (greater than) Percentage of failed requests.
requests_per_sec avg (average) Numerical value with up to two decimal places. > (greater than)
< (less than)
Number of requests per second.
requests count Integer value. > (greater than)
< (less than)
Total number of requests.

Define load test fail criteria

In this section, you configure test criteria for a load test in the Azure portal.

  1. In the Azure portal, go to your Azure Load Testing resource.

  2. On the left pane, select Tests to view the list of load tests.

  3. Select your load test from the list, and then select Edit.

    Screenshot of the list of tests for an Azure load testing resource in the Azure portal, highlighting the 'Edit' button.

  4. On the Test criteria pane, fill the Metric, Aggregate function, Condition, and Threshold values for your test.

    Screenshot of the 'Test criteria' pane for a load test in the Azure portal and highlights the fields for adding a test criterion.

    Optionally, enter the Request name information to add a test criterion for a specific JMeter request. The value should match the name of the JMeter sampler in the JMX file.

    Screenshot of the JMeter user interface, highlighting the request name.

  5. Select Apply to save the changes.

    When you now run the load test, Azure Load Testing uses the test criteria to determine the status of the load test run.

  6. Run the test and view the status in the load test dashboard.

    The dashboard shows each of the test criteria and their status. The overall test status will be failed if at least one criterion was met.

    Screenshot that shows the test criteria on the load test dashboard.

Next steps