Edit

Tutorial: Identify performance regressions by automating load tests with CI/CD

  • Article

This tutorial describes how to quickly identify performance regressions by using Azure Load Testing and CI/CD tools. Quickly identify when your application experiences degraded performance under load by running load tests in Azure Pipelines or GitHub Actions.

In this tutorial, you'll set up a CI/CD pipeline that runs a load test for a sample application on Azure. You'll verify the application behavior under load directly from the CI/CD dashboard. You'll then use load test fail criteria to get alerted when the application doesn't meet your quality requirements.

In this tutorial, you'll use a sample Node.js application and JMeter script. The tutorial doesn't require any coding or Apache JMeter skills.

You'll learn how to:

  • Set up the sample application GitHub repository.
  • Configure service authentication for your CI/CD workflow.
  • Configure the CI/CD workflow to run a load test.
  • View the load test results in the CI/CD dashboard.
  • Define load test fail criteria to identify performance regressions.

Note

Azure Pipelines has a 60-minute timeout on jobs that are running on Microsoft-hosted agents for private projects. If your load test is running for more than 60 minutes, you'll need to pay for additional capacity. If not, the pipeline will time out without waiting for the test results. You can view the status of the load test in the Azure portal.

Prerequisites

  • An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
  • If you're using Azure Pipelines, an Azure DevOps organization and project. If you don't have an Azure DevOps organization, you can create one for free. If you need help with getting started with Azure Pipelines, see Create your first pipeline.
  • A GitHub account, where you can create a repository. If you don't have one, you can create one for free.

Set up the sample application repository

To get started with this tutorial, you first need to set up a sample Node.js web application. The sample application contains an Azure Pipelines definition to deploy the application on Azure and trigger a load test.

  1. Open a browser and go to the sample application's source GitHub repository.

    The sample application is a Node.js app that consists of an Azure App Service web component and an Azure Cosmos DB database.

  2. Select Fork to fork the sample application's repository to your GitHub account.

    Screenshot that shows the button to fork the sample application's GitHub repo.

The sample application's repo contains an Apache JMeter script named SampleApp.jmx. The script tests the three APIs in the sample application:

  • add: Performs an insert on Azure Cosmos DB to store the number of web app visitors.
  • get: Performs a read operation on Azure Cosmos DB to retrieve the visitor count.
  • lasttimestamp: Updates the in-memory time stamp of the last user visit.

Configure service authentication

Before you configure the CI/CD pipeline to run a load test, you'll grant the CI/CD workflow the permissions to access your Azure load testing resource.

To access your Azure Load Testing resource from the Azure Pipelines workflow, you first create a service connection in your Azure DevOps project. The service connection creates an Azure Active Directory service principal. This service principal represents your Azure Pipelines workflow in Azure Active Directory.

Next, you grant permissions to this service principal to create and run a load test with your Azure Load Testing resource.

Create a service connection in Azure Pipelines

Create a service connection in Azure Pipelines so that your CI/CD workflow has access to your Azure subscription. In a next step, you'll then grant permissions to create and run load tests.

  1. Sign in to your Azure DevOps organization (https://dev.azure.com/<your-organization>), and select your project.

  2. Select Project settings > Service connections.

  3. Select + New service connection, select the Azure Resource Manager service connection, and then select Next.

  4. Select the Service Principal (automatic) authentication method, and then select Next.

  5. Enter the service connection information, and then select Save to create the service connection.

    Field Value
    Scope level Subscription.
    Subscription Select the Azure subscription that will host your load testing resource.
    Resource group Leave empty. The pipeline creates a new resource group for the Azure Load Testing resource.
    Service connection name Enter a unique name for the service connection. You'll use this name later, to configure the pipeline definition.
    Grant access permission to all pipelines Checked.

  6. Select the service connection that you created from the list, and then select Manage Service Principal.

    Screenshot that shows selections for managing a service principal.

  7. In the Azure portal, copy the Application (Client) ID value.

Launch Azure Cloud Shell

The Azure Cloud Shell is a free interactive shell that you can use to run the steps in this article. It has common Azure tools preinstalled and configured to use with your account.

To open the Cloud Shell, just select Try it from the upper right corner of a code block. You can also launch Cloud Shell in a separate browser tab by going to https://shell.azure.com.

When Cloud Shell opens, verify that Bash is selected for your environment. Subsequent sessions will use Azure CLI in a Bash environment, Select Copy to copy the blocks of code, paste it into the Cloud Shell, and press Enter to run it.

Sign in to Azure

Cloud Shell is automatically authenticated under the initial account signed-in with. Use the following script to sign in using a different subscription, replacing <Subscription ID> with your Azure Subscription ID. If you don't have an Azure subscription, create an Azure free account before you begin.

subscription="<subscriptionId>" # add subscription here

az account set -s $subscription # ...or use 'az login'

For more information, see set active subscription or log in interactively

Grant access to Azure Load Testing

To grant access to your Azure Load Testing resource, assign the Load Test Contributor role to the service principal. This role grants the service principal access to create and run load tests with your Azure Load Testing service. Learn more about managing users and roles in Azure Load Testing.

  1. Retrieve the ID of the service principal object using the Azure CLI. Replace the text placeholder <application-client-id> with the value you copied.

    object_id=$(az ad sp show --id "<application-client-id>" --query "id" -o tsv)
echo $object_id

  2. Assign the Load Test Contributor role to the service principal:

    subscription=$(az account show --query "id" -o tsv)
echo $subscription

az role assignment create --assignee $object_id \
    --role "Load Test Contributor" \
    --scope /subscriptions/$subscription \
    --subscription $subscription

Configure the CI/CD workflow to run a load test

You'll now create a CI/CD workflow to create and run a load test for the sample application. The sample application repository already contains a CI/CD workflow definition that first deploys the application to Azure, and then creates a load test based on JMeter test script (SampleApp.jmx). You'll update the sample workflow definition file to specify the Azure subscription and application details.

On the first CI/CD workflow run, it creates a new Azure Load Testing resource in your Azure subscription by using the ARMTemplate/template.json Azure Resource Manager (ARM) template. Learn more about ARM templates.

You'll create a new Azure pipeline that is linked to your fork of the sample application repository. This repository contains the following items:

  • The sample application source code.
  • The azure-pipelines.yml pipeline definition file.
  • The SampleApp.jmx JMeter test script.
  • The SampleApp.yaml Azure Load Testing configuration file.

To create and run the load test, the Azure Pipelines definition uses the Azure Load Testing task extension from the Azure DevOps Marketplace.

  1. Open the Azure Load Testing task extension in the Azure DevOps Marketplace, and select Get it free.

  2. Select your Azure DevOps organization, and then select Install to install the extension.

    If you don't have administrator privileges for the selected Azure DevOps organization, select Request to request an administrator to install the extension.

  3. In your Azure DevOps project, select Pipelines in the left navigation, and then select Create pipeline.

  4. On the Connect tab, select GitHub.

  5. Select Authorize Azure Pipelines to allow Azure Pipelines to access your GitHub account for triggering workflows.

  6. On the Select tab, select the sample application's forked repository.

    Screenshot that shows how to select the sample application's GitHub repository.

    Azure Pipelines automatically detects the azure-pipelines.yml pipeline definition file.

  7. Notice that the pipeline definition contains the LoadTest stage, which has two tasks.

    The AzureResourceManagerTemplateDeployment task deploys a new Azure load testing resource in your Azure subscription.

    Next, the AzureLoadTest Azure Load Testing task creates and starts a load test. This task uses the SampleApp.yaml load test configuration file, which contains the configuration parameters for the load test, such as the number of parallel test engines.

    - task: AzureLoadTest@1
  inputs:
    azureSubscription: $(serviceConnection)
    loadTestConfigFile: 'SampleApp.yaml'
    resourceGroup: $(loadTestResourceGroup)
    loadTestResource: $(loadTestResource)
    env: |
      [
        {
        "name": "webapp",
        "value": "$(webAppName).azurewebsites.net"
        }
      ]

    If a load test already exists, the AzureLoadTest task won't create a new load test, but will add a test run to this load test. To identify regressions over time, you can then compare multiple test runs.

  8. On the Review tab, replace the following placeholder text at the beginning of the pipeline definition:

    These variables are used to configure the deployment of the sample application, and to create the load test.

    Placeholder Value
    <Name of your webapp> The name of the Azure App Service web app.
    <Name of your webARM Service connection> The name of the service connection that you created in the previous section.
    <Azure subscriptionId> Your Azure subscription ID.
    <Name of your load test resource> The name of your Azure Load Testing resource.
    <Name of your load test resource group> The name of the resource group that contains the Azure Load Testing resource.

    Screenshot that shows the Azure Pipelines Review tab when you're creating a pipeline.

  9. Select Save and run, enter text for Commit message, and then select Save and run.

    Azure Pipelines now runs the CI/CD workflow and will deploy the sample application and create the load test.

  10. Select Pipelines in the left navigation, and then select new pipeline run from the list to monitor the status.

    You can view the detailed run log by selecting the pipeline job.

    Screenshot that shows how to view pipeline job details.

View load test results

Azure Load Testing enables you to view the results of the load test run directly in the CI/CD workflow output. The CI/CD log contains the following client-side metrics:

  • Response time metrics: average, minimum, median, maximum, and 90-95-99 percentiles.
  • Number of requests per second.
  • Total number of requests.
  • Total number of errors.
  • Error rate.

In addition, the load test results file is available as a workflow run artifact, which you can download for further reporting.

  1. In your Azure DevOps project, select Pipelines, and then select your pipeline definition from the list.

  2. Select the pipeline run to view the run summary.

    Screenshot that shows the pipeline run summary.

  3. Select Load Test in the Jobs section to view the pipeline log.

    Screenshot that shows the Azure Pipelines run log.

    After the load test finishes, you can view the test summary information and the client-side metrics in the pipeline log. The log also shows the URL to go to the Azure Load Testing dashboard for this load test.

  4. In the pipeline log view, select Load Test, and then select 1 artifact produced to download the result files for the load test.

    Screenshot that shows how to download the load test results.

Define test fail criteria

Azure Load Testing enables you to define load test fail criteria. These criteria determine when a load test should pass or fail. For example, your load test should fail when the average response time is greater than a specific value, or when too many errors occur.

When you run a load test as part of a CI/CD pipeline, the status of the pipeline run will reflect the status of the load test. This approach allows you to quickly identify performance regressions, or degraded application behavior when the application is experiencing high load.

In this section, you'll configure test fail criteria based on the average response time and the error rate.

You can specify load test fail criteria for Azure Load Testing in the test configuration YAML file. Learn more about configuring load test fail criteria.

  1. Edit the SampleApp.yml file in your fork of the sample application GitHub repository.

  2. Add the following snippet at the end of the file:

    failureCriteria: 
    - avg(response_time_ms) > 100
    - percentage(error) > 20

    You've now specified fail criteria for your load test based on the average response time and the error rate. The test will fail if at least one of these conditions is met:

    • The aggregate average response time is greater than 100 ms.
    • The aggregate percentage of errors is greater than 20%.

  3. Commit and push the changes to the main branch of the repository.

    The changes will trigger the CI/CD workflow.

  4. After the test finishes, notice that the CI/CD pipeline run has failed.

    In the CI/CD output log, you find that the test failed because one of the fail criteria was met. The load test average response time was higher than the value that you specified in the fail criteria.

    Screenshot that shows pipeline logs after failed test criteria.

    The Azure Load Testing service evaluates the criteria during the test run. If any of these conditions fails, Azure Load Testing service returns a nonzero exit code. This code informs the CI/CD workflow that the test has failed.

  5. Edit the SampleApp.yml file and change the test's fail criteria to increase the criterion for average response time:

    failureCriteria: 
    - avg(response_time_ms) > 5000
    - percentage(error) > 20

  6. Commit the changes to trigger the CI/CD workflow again.

    After the test finishes, you notice that the load test and the CI/CD workflow run complete successfully.

Clean up resources

Important

You can reuse the Azure Load Testing resource that you created for other Azure Load Testing tutorials and how-to articles.

If you don't plan to use any of the resources that you created, delete them so you don't incur any further charges. If you've deployed the sample application in a different resource group, you might want to repeat the following steps.

To delete resources by using the Azure portal:

  1. Select the menu button in the upper-left corner, and then select Resource groups.

  2. From the list, select the resource group that you created.

  3. Select Delete resource group. Screenshot of the selections to delete a resource group in the Azure portal.

  4. Enter the resource group name. Then select Delete.

To delete resources by using the Azure CLI, enter the following command:

az group delete --name <yourresourcegroup>

Remember, deleting the resource group deletes all of the resources within it.

Next steps

You've now created a CI/CD workflow that uses Azure Load Testing to automate running load tests. By using load test fail criteria, you can set the status of the CI/CD workflow and quickly identify performance and application behavior degradations.