Edit

Share via

Automate tests with Azure Pipelines using classic editor

  • Article

In this article, you'll learn how to set up and run your canvas app tests built in Test Studio using the Azure Pipelines classic editor in Azure DevOps Services.

You can use a public project on GitHub - Microsoft/PowerAppsTestAutomation to:

  • Automate operations of signing in to your application.
  • Open a browser on the build agent and execute a set of test cases and suites.
  • View the status of the test execution in the Azure DevOps pipeline.

Prerequisites

Before you begin, you must complete the following steps:

Step 1 - Fork the PowerAppsTestAutomation project

A fork is a copy of a repository. By forking a repository, you can make changes without affecting the original project.

  1. Sign in to GitHub.

  2. Go to the microsoft/PowerAppsTestAutomation repository. You can also search for microsoft/PowerAppsTestAutomation instead, and then select the repository:

    Search GitHub.

  3. Select Fork:

    Select Fork.

  4. Select where you want to fork:

    Select fork account.

Your forked repository will now be available.

Step 2 - Create Test URL .json file

The Test URL .json file will contain the test suite and test case URLs for validating your app. The app test suite and test case URLs can be retrieved by selecting the Copy play link in Test Studio.

You can find a sample file Samples/TestAutomationURLs.json in the repo you created earlier.

  1. Create a new TestURLs.json file in your repo, or use any other file name.
    The file name and location will be mapped in the pipeline variables later in the document.

  2. Copy the format from the Samples/TestAutomationURLs.json file.

  3. Update the Test URLs section with the tests that you want to validate in your app.

  4. Commit the changes to your repo:

    JSON updated.

Create a pipeline

  1. Sign in to your Azure DevOps instance.

  2. Select an existing project or create a new project.

  3. Select Pipelines in the left menu.

  4. Select Create Pipeline:

    Created pipeline.

  5. Select Use the classic editor:

    Classic editor.

  6. Select GitHub as the source.

  7. If necessary, authorize your GitHub connection using OAuth or using a personal access token:

    Pipeline in GitHub.

  8. If needed, edit the connection name.

  9. Select ... (ellipsis) from the right side of Repository input.

  10. Enter the name of your project on GitHub, and then Select it:

    Select a repo.

  11. Select Continue.

  12. In the Select a template screen, select Empty job:

    Select Empty job.

  13. Save your pipeline.

Add tasks to the pipeline

You'll now add new job tasks and configure the tasks to run the tests from the pipeline in this sequence:

  1. Configure screen resolution using PowerShell.

  2. Restore NuGet packages for the PowerAppsTestAutomation solution.

  3. Build the PowerAppsTestAutomation solution.

  4. Add Visual Studio Tests for Google Chrome.

  5. Add Visual Studio Tests for Mozilla Firefox.

Step 1 - Configure screen resolution using PowerShell

  1. Select + next to Agent job 1.

  2. Search for PowerShell.

  3. Select Add to add a PowerShell task to the job:

    Add PowerShell.

  4. Select the task.
    You can also update the display name to Set Agent Screen Resolution to 1920 x 1080 or similar.

  5. Select Inline as the script type, and enter the following in the script window:

    PowerShell 
    # Set agent screen resolution to 1920x1080 to avoid sizing issues with Portal  
Set-DisplayResolution -Width 1920 -Height 1080 -Force
# Wait 10 seconds  
Start-Sleep -s 10
# Verify Screen Resolution is set to 1920x1080  
Get-DisplayResolution

    Script example.

Step 2 - Restore NuGet packages

  1. Select + next to Agent job 1.

  2. Search for NuGet.

  3. Select Add to add a NuGet task to the job.

  4. Select the task.
    You can also update the display name to Restore NuGet Packages or similar.

  5. Select … (ellipsis) in the Path to solution, packages.config, or project.json configuration field.

  6. Select the PowerAppsTestAutomation.sln solution file.

  7. Select OK:

    NuGet package.

Step 3 - Build the PowerAppsTestAutomation solution

  1. Select + next to Agent job 1.

  2. Search for Visual Studio build.

  3. Select Add to add a Visual Studio build task to the job.

  4. Select the task.
    You can also update the display name to Build Power Apps Test Automation Solution or similar.

  5. Select … (ellipsis) in the Solution configuration field.

  6. Select the PowerAppsTestAutomation.sln solution file.

  7. Select OK.

Step 4 - Add Visual Studio Tests for Google Chrome

  1. Select + next to Agent job 1.

  2. Search for Visual Studio Test.

  3. Select Add to add a Visual Studio Test task to the job.

  4. Select the task.
    You can also update the display name to Run Power Apps Test Automation Tests via $(BrowserTypeChrome) or similar.

  5. Remove the default entries in the Test files text field and add the following:

    **\Microsoft.PowerApps.TestAutomation.Tests\bin\\Debug\Microsoft.PowerApps.TestAutomation.Tests.dll

  6. Enter TestCategory=PowerAppsTestAutomation in the Test filter criteria field.

  7. Select Test mix contains UI tests.

    Test mix contains UI tests.

  8. Select … (ellipsis) in the Settings file field.

  9. Expand the Microsoft.PowerApps.TestAutomation.Tests, select the patestautomation.runsettings file, and then select OK:

    Run settings.

  10. Copy the following in the Override test run parameters field.

    -OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeChrome)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(ChromeWebDriver)"

    Note

    This is where the variables in the pipeline are configured, represented above in the form of $(VariableName).

  11. Enter Run Power Apps Test Automation Tests via $(BrowserTypeChrome) or similar in the Test run title field.

    Test run.

Step 5 - Add Visual Studio Tests for Mozilla Firefox

  1. Right-click the Add Visual Studio Tests for Chrome task and select Clone task(s).

  2. Select the task and update the following areas:

    1. Title: Run Power Apps Test Automation Tests via $(BrowserTypeFirefox)

    2. Override test run parameters

      -OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeFirefox)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(GeckoWebDriver)"

    3. Test Run Title: Run Power Apps Test Automation Tests via $(BrowserTypeFirefox)

Configure pipeline variables

You'll now configure the pipeline variables defined in the tasks you've added earlier.

  1. Select the Variables tab.

  2. Select Add and repeat this step to configure the following variables:

Variable name Variable value
BrowserTypeChrome Chrome
BrowserTypeFirefox Firefox
OnlineUrl https://make.powerapps.com
TestAutomationURLFilePath $(Build.SourcesDirectory)\<test URL file>.json
Note: This is the Test URLs .json file you created earlier.
UsePrivateMode true
OnlineUsername Enter the Microsoft Entra email address of the user context that will sign in to the application. Tests will run under the context of this user account. Ensure this user has the ability to run the app and connect to any data sources the app might connect to.

  1. Select Add and enter OnlinePassword in the variable name.

  2. Check the lock image to make this variable a secret.

    Make variable secret.

  3. Save your pipeline configurations.

Run and analyze tests

To validate that your tests are executing successfully, select Queue and then select Run. Your job will start running.

Run job.

As the job runs, select the job to see a detailed status on each of the tasks running:

Job details.

When the job completes, you can view the high-level job summary, and any errors or warnings. By selecting the Tests tab, you can view specific details on the test cases you've executed.

The following example indicates at least one of our test cases has failed while executing the tests using the Chrome browser:

Chrome - failed.

Select RunTestAutomation test to drill into the details on what test case has failed. In the Attachments tab, you can see the summary of the test execution and which test cases have failed or passed in your test suite:

Attachments tab.

Note

If you execute a test suite, you'll see a summary of test cases passed and failed. If you execute a test case, you'll see specific details on the failure with any trace information, if available.

Known limitations

  • Multifactor authentication isn't supported.

  • Test summary will report a single test result per browser. The test result will contain one or more test cases or test suite results.

  • Any authentication process other tha Microsoft Entra sign-in flow requires customization of the sign-in process in the PowerAppsTestAutomation solution.

See also