Automate tests with Azure Pipelines using classic editor

Article
12/16/2022

In this article, you'll learn how to set up and run your canvas app tests built in Test Studio using the Azure Pipelines classic editor in Azure DevOps Services.

You can use a public project on GitHub - Microsoft/PowerAppsTestAutomation to:

Automate operations of signing in to your application.
Open a browser on the build agent and execute a set of test cases and suites.
View the status of the test execution in the Azure DevOps pipeline.

Prerequisites

Before you begin, you must complete the following steps:

Fork the Microsoft/PowerAppsTestAutomation project on GitHub.

Note

Public forks can’t be made private. If you want to create a private repo, please duplicate the repository.
Create a new Test URL .json file in the repo with the App Test URLs you want to run from the pipeline.

Step 1 - Fork the PowerAppsTestAutomation project

A fork is a copy of a repository. By forking a repository, you can make changes without affecting the original project.

Sign in to GitHub.
Go to the microsoft/PowerAppsTestAutomation repository. You can also search for microsoft/PowerAppsTestAutomation instead, and then select the repository:
Select Fork:
Select where you want to fork:

Your forked repository will now be available.

Step 2 - Create Test URL .json file

The Test URL .json file will contain the test suite and test case URLs for validating your app. The app test suite and test case URLs can be retrieved by selecting the Copy play link in Test Studio.

You can find a sample file Samples/TestAutomationURLs.json in the repo you created earlier.

Create a new TestURLs.json file in your repo, or use any other file name.
The file name and location will be mapped in the pipeline variables later in the document.
Copy the format from the Samples/TestAutomationURLs.json file.
Update the Test URLs section with the tests that you want to validate in your app.
Commit the changes to your repo:

Create a pipeline

Sign in to your Azure DevOps instance.
Select an existing project or create a new project.
Select Pipelines in the left menu.
Select Create Pipeline:
Select Use the classic editor:
Select GitHub as the source.
If necessary, authorize your GitHub connection using OAuth or using a personal access token:
If needed, edit the connection name.
Select ... (ellipsis) from the right side of Repository input.
Enter the name of your project on GitHub, and then Select it:
Select Continue.
In the Select a template screen, select Empty job:
Save your pipeline.

Add tasks to the pipeline

You'll now add new job tasks and configure the tasks to run the tests from the pipeline in this sequence:

Configure screen resolution using PowerShell.
Restore NuGet packages for the PowerAppsTestAutomation solution.
Build the PowerAppsTestAutomation solution.
Add Visual Studio Tests for Google Chrome.
Add Visual Studio Tests for Mozilla Firefox.

Step 1 - Configure screen resolution using PowerShell

Select + next to Agent job 1.
Search for PowerShell.
Select Add to add a PowerShell task to the job:
Select the task.
You can also update the display name to Set Agent Screen Resolution to 1920 x 1080 or similar.

Select Inline as the script type, and enter the following in the script window:

# Set agent screen resolution to 1920x1080 to avoid sizing issues with Portal  
Set-DisplayResolution -Width 1920 -Height 1080 -Force
# Wait 10 seconds  
Start-Sleep -s 10
# Verify Screen Resolution is set to 1920x1080  
Get-DisplayResolution

Script example.

Step 2 - Restore NuGet packages

Select + next to Agent job 1.
Search for NuGet.
Select Add to add a NuGet task to the job.
Select the task.
You can also update the display name to Restore NuGet Packages or similar.
Select … (ellipsis) in the Path to solution, packages.config, or project.json configuration field.
Select the PowerAppsTestAutomation.sln solution file.
Select OK:

Step 3 - Build the PowerAppsTestAutomation solution

Select + next to Agent job 1.
Search for Visual Studio build.
Select Add to add a Visual Studio build task to the job.
Select the task.
You can also update the display name to Build Power Apps Test Automation Solution or similar.
Select … (ellipsis) in the Solution configuration field.
Select the PowerAppsTestAutomation.sln solution file.
Select OK.

Step 4 - Add Visual Studio Tests for Google Chrome

Select + next to Agent job 1.
Search for Visual Studio Test.
Select Add to add a Visual Studio Test task to the job.
Select the task.
You can also update the display name to Run Power Apps Test Automation Tests via $(BrowserTypeChrome) or similar.
Remove the default entries in the Test files text field and add the following:

**\Microsoft.PowerApps.TestAutomation.Tests\bin\\Debug\Microsoft.PowerApps.TestAutomation.Tests.dll
Enter TestCategory=PowerAppsTestAutomation in the Test filter criteria field.
Select Test mix contains UI tests.
Select … (ellipsis) in the Settings file field.
Expand the Microsoft.PowerApps.TestAutomation.Tests, select the patestautomation.runsettings file, and then select OK:

Copy the following in the Override test run parameters field.

-OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeChrome)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(ChromeWebDriver)"

Note

This is where the variables in the pipeline are configured, represented above in the form of $(VariableName).

Enter Run Power Apps Test Automation Tests via $(BrowserTypeChrome) or similar in the Test run title field.

Step 5 - Add Visual Studio Tests for Mozilla Firefox

Right-click the Add Visual Studio Tests for Chrome task and select Clone task(s).

Select the task and update the following areas:

Title: Run Power Apps Test Automation Tests via $(BrowserTypeFirefox)

Override test run parameters

-OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeFirefox)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(GeckoWebDriver)"

Test Run Title: Run Power Apps Test Automation Tests via $(BrowserTypeFirefox)

Configure pipeline variables

You'll now configure the pipeline variables defined in the tasks you've added earlier.

Select the Variables tab.
Select Add and repeat this step to configure the following variables:

Variable name	Variable value
BrowserTypeChrome	Chrome
BrowserTypeFirefox	Firefox
OnlineUrl	https://make.powerapps.com
TestAutomationURLFilePath	`$(Build.SourcesDirectory)\<test URL file>.json` Note: This is the Test URLs .json file you created earlier.
UsePrivateMode	true
OnlineUsername	Enter the Microsoft Entra email address of the user context that will sign in to the application. Tests will run under the context of this user account. Ensure this user has the ability to run the app and connect to any data sources the app might connect to.

Select Add and enter OnlinePassword in the variable name.
Check the lock image to make this variable a secret.
Save your pipeline configurations.

Run and analyze tests

To validate that your tests are executing successfully, select Queue and then select Run. Your job will start running.

Run job.

As the job runs, select the job to see a detailed status on each of the tasks running:

Job details.

When the job completes, you can view the high-level job summary, and any errors or warnings. By selecting the Tests tab, you can view specific details on the test cases you've executed.

The following example indicates at least one of our test cases has failed while executing the tests using the Chrome browser:

Chrome - failed.

Select RunTestAutomation test to drill into the details on what test case has failed. In the Attachments tab, you can see the summary of the test execution and which test cases have failed or passed in your test suite:

Attachments tab.