Azure DevOps: Getting started with JMeter Load Tests
Overview
This article provides how to get started using DevOps Load test and Apache JMeter. This allows for simulating activity for websites and APIs. Being able to perform a load test against a website or API has many benefits. This can be used to determine the effectiveness of routing and load balancing, server performance and latency as well as verifying the functionality of the website or API. This is often included as part of a CI/CD pipeline when a new version of a website or API is deployed.
*This article was written in response to a comment on the Azure On The Cheap post. *
DevOps Load test
The future of DevOps Cloud-based load testing (CLT) looks bleak. Visual Studio 2019 will be the last version of Visual Studio with web performance and load test capability. The ability to perform CLT has been deprecated by the end of March 2020. The capability for running JMeter scripts is still available and it is suspected that one day the capability will disappear. What is not clear is if an alternative in DevOps will become available. Until then, we will enjoy this great capability!
The DevOps Load test feature is located under the Test Plans section of a DevOps project:
This allows for load tests to be managed. For this article, we are concerned about running a JMeter tests.
Scheduling a JMeter test is simple. You are able to specify a test script and supporting files. The location that the load will be generated from as well as the duration the load should run for and the number of agents to run the test can be specified.
This allows for load to be generated from one region and allows for interesting simulations. First, how does the website or API behave with a low level of traffic compared to a high level of traffic. Does it scale up automatically? Does it scale back down when the load test finishes? For a globally distributed website or API, does Azure Front Door or Azure Traffic Manager direct the traffic to the appropriate region?
The following is an example of performing a load test against a globally distributed website. The load was generated in Japan West and involved 5 agents for one minute. Once the load test has been completed a summary of the load test is available.
There are several things to point out. First the total number of requests was 24,452 and all of these were successful. A success of a request is determined by the JMeter script, and in this situation it means they all responded with a status code of 200. The total load generated by the 5 agents was 176 users which averaged 815 requests per second. The usage, 500 VUMs, is what will be billed against the associated DevOps account. Also of interest are the top 5 slowest requests. In this example the call to an API for retrieving data ended up being slowest with an average of .339 milliseconds.
DevOps load testing also provide some charts summarizing the activity.
The most interesting thing with the example is it indicates the Azure Function's startup time. This is because the Azure Function is on a consumption plan and had not had any recent requests. This is reflected in the Performance and throughput charts.
Test errors and general messages are available on the Diagnostics tab.
Note the message about the service being deprecated. We really hope the reason for the service still being available past the date is because something better is about to be previewed!
The last tab of the load test contains information about the agents run including attributes and jmeter.logs.
These logs are especially useful in investigating any failures running the scripts. For example, the most important tip in this article is show below:
The agents only support JMeter 3.2 so we have to author using a compatible JMeter version.
Apache JMeter
The easiest way to get started with JMeter is to download the JMeter binaries. We need to be sure to grab the 3.2 version of the binaries:
Another tip before launching JMeter is to verify the java version. This can be done in a command prompt using the java -version command.
With the correct version of java, it should be as simple as launching JMeter using the JMeter.bat file.
The simplest way to get started is to use a template to record some web activity. Using a Firefox browser, you will use JMeter as a proxy to browse the website.
With a new JMeter project, select templates from the File menu. The template we will use is called Recording. If you don't see this template then double check you are using the 3.2 version.
Note the Https Test Script Recorder in the created project.
Select this node and also note the port of 8888. This will be the port that the proxy is hosted on so we will need to update our network settings in Firefox appropriately. In Firefox, navigate to the proxy settings (each version seems different so check the documentation if you have trouble finding the proxy settings) and set the proxy to be localhost with port 8888:
Back in JMeter, start the proxy by clicking the Start button.
You will then be greeted by an important dialog.
This is worth reading as we need to add a temporary certificate. The certificate is located in the bin directory and follow the instructions on the JMeter website.
Once the certificate has been installed and the JMeter proxy is running, in Firefox navigate to the website to be recorded and perform some steps. Once you are finished, stop the proxy. You will see the recording created in the Recording Controller.
JMeter testing is a huge topic so we will not go into more detail here. Hopefully this is enough to get you started, and more importantly, get you interested in exploring JMeter more.
Once the JMeter script has been crafted, the file can be saved and the same file can be used in DevOps Load tests.
Conclusion
This article provides how to get started using DevOps Load test and Apache JMeter. There are many benefits to performing load tests and we have only scratched the service of load testing and the capabilities of JMeter. The goal of the article was to give enough context about these technologies to generate interest.
It will be a shame to see this capability disappear from DevOps. Please post in the comments alternatives you have found for running load tests.
The following are some helpful resources: