Edit

Tutorial: Identify performance regressions by automating load tests with CI/CD

  • Article

This tutorial describes how to identify performance regressions by using Azure Load Testing and CI/CD tools. Set up a CI/CD workflow in Azure Pipelines to automatically run a load test for your application. Use test fail criteria to get alerted about application changes that affect performance or stability.

With regression testing, you want to validate that code changes don't affect the application functionality, performance, and stability. Azure Load Testing enables you to verify that your application continues to meet your performance and stability requirements when put under real-world user load. Test fail criteria give you a point-in-time check about how the application performs.

In this tutorial, you use a sample Node.js application and JMeter script. The tutorial doesn't require any coding or Apache JMeter skills.

You'll learn how to:

  • Deploy the sample application on Azure.
  • Create a load test by using a JMeter script.
  • Set up a CI/CD workflow from the Azure portal.
  • View the load test results in the CI/CD dashboard.
  • Define load test fail criteria to identify performance regressions.

Note

Azure Pipelines has a 60-minute timeout on jobs that are running on Microsoft-hosted agents for private projects. If your load test is running for more than 60 minutes, you'll need to pay for additional capacity. If not, the pipeline will time out without waiting for the test results. You can view the status of the load test in the Azure portal.

Prerequisites

  • An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
  • An Azure DevOps organization and project. If you don't have an Azure DevOps organization, you can create one for free.

Deploy the sample application

To get started with this tutorial, you first need to set up a sample Node.js web application.

  1. Open Windows PowerShell, sign in to Azure, and set the subscription:

    az login
az account set --subscription <your-Azure-Subscription-ID>

  2. Clone the sample application's source repo:

    git clone https://github.com/Azure-Samples/nodejs-appsvc-cosmosdb-bottleneck.git

    The sample application is a Node.js app that consists of an Azure App Service web component and an Azure Cosmos DB database. The repo includes a PowerShell script that deploys the sample app to your Azure subscription. It also has an Apache JMeter script that you'll use in later steps.

  3. Go to the Node.js app's directory and deploy the sample app by using this PowerShell script:

    cd nodejs-appsvc-cosmosdb-bottleneck
.\deploymentscript.ps1

    Tip

    You can install PowerShell on Linux/WSL or macOS.

    After you install it, you can run the previous command as pwsh ./deploymentscript.ps1.

  4. At the prompt, provide:

    • Your Azure subscription ID.
    • A unique name for your web app.
    • A location. By default, the location is eastus. You can get region codes by running the Get-AzLocation command.

    Important

    For your web app's name, use only lowercase letters and numbers. Don't use spaces or special characters.

  5. After deployment finishes, go to the running sample application by opening https://<yourappname>.azurewebsites.net in a browser window.

Now that you have the application deployed and running, you can create a URL-based load test against it.

Create a load test

Before you set up the CI/CD workflow in Azure Pipelines, you create an Azure load testing resource and create load test by uploading a JMeter test script in the Azure portal. The JMeter script tests three endpoints in the sample application: lasttimestamp, add, and get.

After you create the load test, you can then set up the CI/CD workflow from the Azure portal.

Create the Azure load testing resource

The Azure load testing resource is a top-level resource for your load-testing activities. This resource provides a centralized place to view and manage load tests, test results, and related artifacts.

If you already have a load testing resource, skip this section and continue to Create a load test by uploading a JMeter script.

If you don't yet have an Azure load testing resource, create one now:

  1. Sign in to the Azure portal by using the credentials for your Azure subscription.

  2. From the Azure portal menu or the Home page, select Create a resource.

  3. On the Marketplace page, search for and select Azure Load Testing.

  4. On the Azure Load Testing pane, select Create.

  5. On the Create a load testing resource page, enter the following information:

    Field Description
    Subscription Select the Azure subscription that you want to use for this Azure Load Testing resource.
    Resource group Select an existing resource group. Or select Create new, and then enter a unique name for the new resource group.
    Name Enter a unique name to identify your Azure Load Testing resource.
    The name can't contain special characters, such as \/""[]:|<>+=;,?*@&, or whitespace. The name can't begin with an underscore (_), and it can't end with a period (.) or a dash (-). The length must be 1 to 64 characters.
    Location Select a geographic location to host your Azure Load Testing resource.
    This location also determines where the test engines are hosted and where the JMeter client requests originate from.

    Note

    Optionally, you can configure more details on the Tags tab. Tags are name/value pairs that enable you to categorize resources and view consolidated billing by applying the same tag to multiple resources and resource groups.

  6. After you're finished configuring the resource, select Review + Create.

  7. Review the settings you provide, and then select Create. It takes a few minutes to create the account. Wait for the portal page to display Your deployment is complete before moving on.

  8. To view the new resource, select Go to resource.

    Screenshot that shows the deployment completion screen.

  9. Optionally, manage access to your Azure Load Testing resource.

    Azure Load Testing uses role-based access control (RBAC) to manage permissions for your resource. If you encounter this message, your account doesn't have the necessary permissions to manage tests.

    Screenshot that shows an error message in the Azure portal that you're not authorized to use the Azure Load Testing resource.

Create a load test by uploading a JMeter script

You can create a load test by uploading an Apache JMeter test script. The test script defines the test plan, and describes the application requests to invoke and any custom logic for the load test. Azure Load Testing abstracts the infrastructure for running the test script at scale.

To create a load test by uploading a JMeter script in the Azure portal:

  1. Sign in to the Azure portal by using the credentials for your Azure subscription.

  2. Go to your Azure Load Testing resource, select Tests from the left pane, select + Create, and then select Upload a JMeter script.

    Screenshot that shows the Azure Load Testing page and the button for creating a new test.

  3. On the Basics tab, enter the Test name and Test description information.

    Screenshot that shows the Basics tab for creating a test.

  4. On the Test plan tab, select the sample application JMeter script, and then select Upload to upload the file to Azure.

    You can find the JMeter script SampleApp.jmx in the repository you cloned earlier.

    Screenshot that shows the Test plan tab.

  5. On the Parameters tab, add an environment variable for the sample application endpoint:

    The test script uses an environment variable to retrieve the endpoint of the sample application.

    Field Value
    Name webapp
    Value Hostname of the deployed sample application, without https:// prefix.

    Screenshot that shows the Parameters plan tab, highlighting the environment variable for the sample app hostname.

  6. Select Review + Create, review the values, and then select Create to create and run the load test.

    Note

    After creating the load test, it might take a few minutes for the load test to finish running.

Set up the CI/CD workflow from the Azure portal

Now that you have load testing resource and a load test for the sample application, you can set up a new CI workflow to automatically run your load test. Azure Load Testing enables you to set up a new CI workflow in Azure Pipelines from the Azure portal.

Create the CI/CD workflow

  1. In the Azure portal, go to your Azure load testing resource.

  2. On the left pane, select Tests to view the list of tests.

  3. Select the test you created previously by selecting the checkbox, and then select Set up CI/CD.

    Screenshot that shows the list of tests in Azure portal.

  4. Enter the following details for creating a CI/CD pipeline definition:

    Setting Value
    Organization Select the Azure DevOps organization where you want to run the pipeline from.
    Project Select the project from the organization selected previously.
    Repository Select the source code repository to store and run the Azure pipeline from.
    Branch Select the branch in the selected repository.
    Repository branch folder (Optional) Enter the repository branch folder name in which you'd like to commit. If empty, the root folder is used.
    Override existing files Check this setting.
    Service connection Select Create new to create a new service connection to allow Azure Pipelines to connect to the load testing resource.

    Screenshot that shows the settings to be configured to set up a CI/CD pipeline.

    Important

    If you're getting an error creating a PAT token, or you don't see any repositories, make sure to connect your Azure DevOps organization to Microsoft Entra ID. Make sure the directory in Azure DevOps matches the directory you're using for Azure Load Testing. After connecting to Microsoft Entra ID, close and reopen your browser window.

  5. Select Create Pipeline to start creating the pipeline definition.

    Azure Load Testing performs the following actions to configure the pipeline:

    • Create a new service connection of type Azure Resource Manager in the Azure DevOps project. The service principal is automatically assigned the Load Test Contributor role on the Azure load testing resource.

    • Commit the JMeter script and test configuration YAML to the source code repository.

    • Create a pipeline definition that invokes the Azure load testing resource and runs the load test.

  6. When the pipeline creation finishes, you receive a notification in the Azure portal with a link to the pipeline.

Run the CI/CD workflow

You can now manually trigger the CI/CD workflow to validate that the load test is run correctly.

  1. Sign in to your Azure DevOps organization (https://dev.azure.com/<your-organization>), and select your project.

    Replace the <your-organization> text placeholder with your project identifier.

  2. Select Pipelines in the left navigation

    Notice that there's a new pipeline created in your project.

    Screenshot that shows the Azure Pipelines page, showing the pipeline that Azure Load Testing generated.

  3. Select the pipeline, select Run pipeline, and then select Run to start the CI workflow.

    The first time you run the pipeline, you need to grant the pipeline permission to access the service connection and connect to Azure. Until you grant permission, the CI workflow run remains in the waiting state.

    Screenshot that shows the Azure Pipelines 'Run pipeline' page.

  4. Select the Load Test job to view the job details.

    An alert message is shown that the pipeline needs permission to access a resource.

    Screenshot that shows the Azure Pipelines run details page, showing a warning that the pipeline needs additional permissions.

  5. Select View > Permit > Permit to grant the permission.

    The CI/CD pipeline run now starts and runs your load test.

You've now configured and run an Azure Pipelines workflow that automatically runs a load test each time a source code update is made.

View load test results

While the CI pipeline is running, you can view the load test statistics directly in the Azure Pipelines log. The CI/CD log displays the following load test statistics: response time metrics, requests per second, total number of requests, number of errors, and error rate. Alternately, you can navigate directly to the load test dashboard in the Azure portal by selecting the URL in the pipeline log.

Screenshot that shows the Azure Pipelines run log, displaying the load testing metrics and Azure portal link.

You can also download the load test results file, which is available as a pipeline artifact. In the pipeline log view, select Load Test, and then select 1 artifact produced to download the result files for the load test.

Screenshot that shows how to download the load test results.

Add test fail criteria

To identify performance regressions, you can analyze the test metrics for each pipeline run logs. Ideally, you want the pipeline run to fail whenever your performance or stability requirements aren't met.

Azure Load Testing enables you to define load test fail criteria based on client-side metrics, such as the response time or error rate. When at least one of the fail criteria isn't met, the status of the CI pipeline is set to failed accordingly. With test fail criteria, you can now quickly identify if a specific application build results in a performance regression.

To define test fail criteria for the average response time and the error rate:

  1. In your Azure DevOps project, select Repos > Files.

  2. Select the alt-config-<unique_id>.yml file, and then select Edit.

    This YAML file specifies the load test configuration settings, such as the reference to the JMeter test script, the list of fail criteria, references to input data files, and more.

  3. Replace the failureCriteria: with the following snippet to define two test fail criteria:

    failureCriteria: 
    - avg(response_time_ms) > 100
    - percentage(error) > 20

    Screenshot that shows how to update the load test configuration file with test criteria in Azure Pipelines.

    You've now specified fail criteria for your load test based on the average response time and the error rate. The test fails if at least one of these conditions is met:

    • The aggregate average response time is greater than 100 ms.
    • The aggregate percentage of errors is greater than 20%.

  4. Select Commit to save the updates.

    Updating the file will trigger the CI/CD workflow.

  5. After the test finishes, notice that the CI/CD pipeline run has failed.

    In the CI/CD output log, you find that the test failed because one of the fail criteria was met. The load test average response time was higher than the value that you specified in the fail criteria.

    Screenshot that shows pipeline logs after failed test criteria.

    The Azure Load Testing service evaluates the criteria during the test run. If any of these conditions fails, Azure Load Testing service returns a nonzero exit code. This code informs the CI/CD workflow that the test has failed.

  6. Edit the alt-config-<unique_id>.yml file and change the test's fail criteria to increase the criterion for average response time:

    failureCriteria: 
    - avg(response_time_ms) > 5000
    - percentage(error) > 20

  7. Commit the changes to trigger the CI/CD workflow again.

    After the test finishes, you notice that the load test and the CI/CD workflow run complete successfully.

Clean up resources

Important

You can reuse the Azure Load Testing resource that you created for other Azure Load Testing tutorials and how-to articles.

If you don't plan to use any of the resources that you created, delete them so you don't incur any further charges. If you've deployed the sample application in a different resource group, you might want to repeat the following steps.

To delete resources by using the Azure portal:

  1. Select the menu button in the upper-left corner, and then select Resource groups.

  2. From the list, select the resource group that you created.

  3. Select Delete resource group. Screenshot of the selections to delete a resource group in the Azure portal.

  4. Enter the resource group name. Then select Delete.

To delete resources by using the Azure CLI, enter the following command:

az group delete --name <yourresourcegroup>

Remember, deleting the resource group deletes all of the resources within it.

Related content

In this tutorial, you've set up a new CI/CD workflow in Azure Pipelines to automatically run a load test with every code change. By using test fail criteria, you can identify when a performance regression was introduced in the application.