Quickstart: Set up continuous end-to-end testing with Microsoft Playwright Testing Preview

In this quickstart, you set up continuous end-to-end testing with Microsoft Playwright Testing Preview to validate that your web app runs correctly across different browsers and operating systems with every code commit. Learn how to add your Playwright tests to a continuous integration (CI) workflow, such as GitHub Actions, Azure Pipelines, or other CI platforms.

After you complete this quickstart, you have a CI workflow that runs your Playwright test suite at scale with Microsoft Playwright Testing.

Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

  • A GitHub account. If you don't have a GitHub account, you can create one for free.
  • A GitHub repository that contains your Playwright test specifications and GitHub Actions workflow. To create a repository, see Creating a new repository.
  • A GitHub Actions workflow. If you need help with getting started with GitHub Actions, see create your first workflow

Configure a service access token

Microsoft Playwright Testing uses access tokens to authorize users to run Playwright tests with the service. You can generate a service access token in the Playwright portal, and then specify the access token in the service configuration file.

To generate an access token and store it as a CI workflow secret, perform the following steps:

  1. Sign in to the Playwright portal with your Azure account.

  2. Select the workspace settings icon, and then go to the Access tokens page.

    Screenshot that shows the access tokens settings page in the Playwright Testing portal.

  3. Select Generate new token to create a new access token for your CI workflow.

  4. Enter the access token details, and then select Generate token.

    Screenshot that shows setup guide in the Playwright Testing portal, highlighting the 'Generate token' button.

    Screenshot that shows how to copy the generated access token in the Playwright Testing portal.

  5. Store the access token in a CI workflow secret to avoid specifying the token in clear text in the workflow definition:

    1. Go to your GitHub repository, and select Settings > Secrets and variables > Actions.

    2. Select New repository secret.

    3. Enter the secret details, and then select Add secret to create the CI/CD secret.

      Parameter Value
      Name PLAYWRIGHT_SERVICE_ACCESS_TOKEN
      Value Paste the workspace access token you copied previously.
    4. Select OK to create the workflow secret.

Get the service region endpoint URL

In the service configuration, you have to provide the region-specific service endpoint. The endpoint depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL and store it as a CI workflow secret, perform the following steps:

  1. Sign in to the Playwright portal with your Azure account.

  2. On the workspace home page, select View setup guide.

    Tip

    If you have multiple workspaces, you can switch to another workspace by selecting the workspace name at the top of the page, and then select Manage all workspaces.

  3. In Add region endpoint in your setup, copy the service endpoint URL.

    The endpoint URL matches the Azure region that you selected when creating the workspace.

  4. Store the service endpoint URL in a CI workflow secret:

    Secret name Value
    PLAYWRIGHT_SERVICE_URL Paste the endpoint URL you copied previously.

Add service configuration file

If you haven't configured your Playwright tests yet for running them on cloud-hosted browsers, add a service configuration file to your repository. In the next step, you then specify this service configuration file on the Playwright CLI.

  1. Create a new file playwright.service.config.ts alongside the playwright.config.ts file.

    Optionally, use the playwright.service.config.ts file in the sample repository.

  2. Add the following content to it:

    /*
    * This file enables Playwright client to connect to remote browsers.
    * It should be placed in the same directory as playwright.config.ts.
    */
    
    import { defineConfig } from '@playwright/test';
    import config from './playwright.config';
    import dotenv from 'dotenv';
    
    // Define environment on the dev box in .env file:
    //  .env:
    //    PLAYWRIGHT_SERVICE_ACCESS_TOKEN=XXX
    //    PLAYWRIGHT_SERVICE_URL=XXX
    
    // Define environment in your GitHub workflow spec.
    //  env:
    //    PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
    //    PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
    //    PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
    
    dotenv.config();
    
    // Name the test run if it's not named yet.
    process.env.PLAYWRIGHT_SERVICE_RUN_ID = process.env.PLAYWRIGHT_SERVICE_RUN_ID || new Date().toISOString();
    
    // Can be 'linux' or 'windows'.
    const os = process.env.PLAYWRIGHT_SERVICE_OS || 'linux';
    
    export default defineConfig(config, {
      // Define more generous timeout for the service operation if necessary.
      // timeout: 60000,
      // expect: {
      //   timeout: 10000,
      // },
      workers: 20,
    
      // Enable screenshot testing and configure directory with expectations.
      // https://learn.microsoft.com/azure/playwright-testing/how-to-configure-visual-comparisons
      ignoreSnapshots: false,
      snapshotPathTemplate: `{testDir}/__screenshots__/{testFilePath}/${os}/{arg}{ext}`,
      
      use: {
        // Specify the service endpoint.
        connectOptions: {
          wsEndpoint: `${process.env.PLAYWRIGHT_SERVICE_URL}?cap=${JSON.stringify({
            // Can be 'linux' or 'windows'.
            os,
            runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID
          })}`,
          timeout: 30000,
          headers: {
            'x-mpt-access-key': process.env.PLAYWRIGHT_SERVICE_ACCESS_TOKEN!
          },
          // Allow service to access the localhost.
          exposeNetwork: '<loopback>'
        }
      },
      // Tenmp workaround for config merge bug in OSS https://github.com/microsoft/playwright/pull/28224
      projects: config.projects? config.projects : [{}]
    });
    
  3. Save and commit the file to your source code repository.

Update the workflow definition

Update the CI workflow definition to run your Playwright tests with the Playwright CLI. Pass the service configuration file as an input parameter for the Playwright CLI. You configure your environment by specifying environment variables.

  1. Open the CI workflow definition

  2. Add the following steps to run your Playwright tests in Microsoft Playwright Testing.

    The following steps describe the workflow changes for GitHub Actions or Azure Pipelines. Similarly, you can run your Playwright tests by using the Playwright CLI in other CI platforms.

    - name: Install dependencies
      working-directory: path/to/playwright/folder # update accordingly
      run: npm ci
    
    - name: Run Playwright tests
      working-directory: path/to/playwright/folder # update accordingly
      env:
        # Access token and regional endpoint for Microsoft Playwright Testing
        PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
        PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
        PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
      run: npx playwright test -c playwright.service.config.ts --workers=20
    
    - name: Upload Playwright report
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: playwright-report
        path: path/to/playwright/folder/playwright-report/ # update accordingly
        retention-days: 10
    
  3. Save and commit your changes.

    When the CI workflow is triggered, your Playwright tests will run in your Microsoft Playwright Testing workspace on cloud-hosted browsers, across 20 parallel workers.

Caution

With Microsoft Playwright Testing, you get charged based on the number of total test minutes. If you're a first-time user or getting started with a free trial, you might start with running a single test at scale instead of your full test suite to avoid exhausting your free test minutes.

After you validate that the test runs successfully, you can gradually increase the test load by running more tests with the service.

You can run a single test with the service by using the following command-line:

npx playwright test {name-of-file.spec.ts} --config=playwright.service.config.ts

Enable test results reporting

Microsoft Playwright Testing now supports viewing test results in the Playwright Portal. During preview access is only available by invitation only.

Important

The reporting feature of Microsoft Playwright Testing service is free of charge during the invite-only preview. However, existing functionality of cloud-hosted browsers continues to bill per the Azure pricing plan.

Once you have access to the reporting tool, use the following steps to set up your tests.

  1. From the workspace home page, navigate to Settings.

    Screenshot that shows settings selection for a workspace in the Playwright Testing portal.

  2. From Settings, select General and make sure reporting is Enabled.

    Screenshot that shows how to enable reporting for a workspace in the Playwright Testing portal.

  3. Create a GitHub Personal Access Token by following these steps.

    You need to provide read:packages permissions to the token. This token is referred to as PAT_TOKEN_PACKAGE for the rest of this article.

  4. Store the GitHub token in a CI workflow secret to avoid specifying the token in clear text in the workflow definition:

    1. Go to your GitHub repository, and select Settings > Secrets and variables > Actions.

    2. Select New repository secret.

    3. Enter the secret details, and then select Add secret to create the CI/CD secret.

      Parameter Value
      Name PAT_TOKEN_PACKAGE
      Value Paste the workspace access token you copied previously.
    4. Select OK to create the workflow secret.

  5. Update package.json file with the package.

        "dependencies": {
            "@microsoft/mpt-reporter": "0.1.0-22052024-private-preview"
        }
    
  6. Update the Playwright config file.

    Add Playwright Testing reporter to Playwright.config.ts in the same way you use other reporters.

    import { defineConfig } from '@playwright/test';
    
    export default defineConfig({
        reporter: [
        ['list'],
        ['json', {  outputFile: 'test-results.json' }],
        ['@microsoft/mpt-reporter'] // Microsoft Playwright Testing reporter
        ],
    });
    

    Make sure that the artifacts are enabled in the config for better troubleshooting.

    use: {
        // ...
        trace: 'on-first-retry',
        video:'retain-on-failure',
        screenshot:'only-on-failure',
        }
    
  7. Update the CI workflow definition to install the reporting package before running the tests to publish the report of your Playwright tests in Microsoft Playwright Testing.

    - name: Install reporting package
      working-directory: path/to/playwright/folder # update accordingly
      run: |
        npm config set @microsoft:registry=https://npm.pkg.github.com
        npm set //npm.pkg.github.com/:_authToken ${{secrets.PAT_TOKEN_PACKAGE}} 
        npm install
    
    - name: Run Playwright tests
      working-directory: path/to/playwright/folder # update accordingly
      env:
        # Access token and regional endpoint for Microsoft Playwright Testing
        PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
        PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
        PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
      run: npx playwright test
    

    Tip

    You can use Microsoft Playwright Testing service to publish test results to the portal independent of the cloud-hosted browsers feature.

You've successfully set up a continuous end-to-end testing workflow to run your Playwright tests at scale on cloud-hosted browsers.