Silent testing framework

The goal of this document is to provide technical teams that manage corporate webcasting platforms instructions on how to use the Microsoft eCDN Silent Testing tool to audit their corporate network(s) in preparation for actual events.

The Microsoft eCDN Silent Testing framework allows simulations to be run on multiple devices easily in order to emulate and examine how a given network behaves under the load of a video event.

A silent test is a real video session that runs in the background (muted) on an end-user device. The user can keep working on their computer without knowing that the test is running, though a slowdown in network connectivity respective to the video's bandwidth might occur.

The framework comprises three main components:

• Runner Devices
• Management Dashboard
• Analytics

These components are explained one by one in the following sections.

Runner devices

Each device that connects to the framework is considered a "runner". Each runner simulates a single viewer and communicates with Microsoft eCDN's backend to get instructions on which test should be executed. Most of the time there are no tests executing, in which case the runner waits idle until a test starts. Instead of deploying a designated agent application on every computer to act as a runner, Microsoft eCDN leverages existing software that's commonly installed on end-user machines to launch runners in the background.

Since the runner is essentially a webpage, it can be opened in any browser or browser-like environment. There are two ways of instantiating a runner.

Important

Microsoft Edge or Google Chrome must be installed on the end-user's machine. Additionally, the device must be turned on and connected to the internet to participate in silent tests.

Direct Runner

Manually opening the runner page with the following URL, taking care to replace the placeholder TENANT_ID_HERE with your tenant's ID in a browser, is considered a "direct runner".

https://st-sdk.ecdn.teams.microsoft.com/?customerId=TENANT_ID_HERE&adapterId=Direct

Caution

Whereas a Silent Runner is instanced with the necessary arguments to expose the machine's IP address to the Microsoft eCDN service, a Direct Runner uses the machine's global settings. As such, if you haven't already disabled mDNS IP obfuscation, the Direct Runner is unlikely to peer.

Silent Runner

We provide PowerShell & Bash scripts that launch a headless Chromium browser in the background with the runner page, which is considered a "silent runner". The script can then be executed on a group of users to connect them to the framework.

For more information, see Appendix B: Integrating Runners Using Headless Browser

Management dashboard

The management dashboard allows tests to be scheduled, modified, and canceled, and also shows the number of connected runners. The main window lists pending tests, ongoing tests and past tests that have already ended. Completed tests are shown for 24 hours and are then hidden from the list.

Important

The number of devices or "Assigned Clients" that execute a scheduled test depends on the number of online runners at the time of the test's scheduled start, not at the time of its scheduling.

Analytics

When a test is scheduled, it's set to "pending" mode. Once the start time is reached, the test is activated and all of the online runners are sent an activation signal. The target page is then launched by each runner, and (muted) video starts playing in the hidden window. Microsoft eCDN SDK collects network and UX metrics that are presented in various charts and graphs available in the Analytics dashboard. Analytics are reported while the test is running so admins can check the status even before the test ends

Concurrency

The Concurrency graph shows the number of active users over time. To be considered active, a user must be playing video.

HTTP + P2P speed

The network throughput graph shows a breakdown of network consumption in HTTP and P2P.

Represented as	Description	Axis
Dark blue bars	HTTP bandwidth	left
Orange bars	P2P bandwidth	left
Green dotted line	Ratio of P2P out of the total as a percentage	right

For example, a P2P ratio of 90% means that only 10% of the traffic was downloaded via HTTP and the rest was peered between users.

If P2P is lower than expected, it means that either the user concurrency wasn't high enough, or the network requires more optimization. For troubleshooting, see the Troubleshooting low peering efficiency documentation.

User experience

The user experience graph shows the combined time spent playing vs time spent rebuffering (frozen video).

Represented as	Description	Axis
Green bars	Aggregate time spent playing in minutes	left
Red bars	Combined time spent rebuffering in minutes	left
Blue dotted line	Ratio of rebuffering out of the total time as a percentage	right

For example, a 2% rebuffering ratio means that video was playing correctly for 98% of the time, whereas for 2% of the time the video was stuck.

Rebuffering should ideally be below 1%. High numbers or spikes in rebuffering might suggest network congestion, server overload or misconfigured content.

Network requirements

The Silent testing framework uses the following domains and ports:

Hostname	Ports	Protocol	Description
*.ecdn.teams.microsoft.com	443	HTTPS	Runner page & resources
*.ecdn.teams.microsoft.com	443	WSS	WebSocket connection to Microsoft eCDN's backend
any	High ports 10,000 +	SCTP	This is required by WebRTC peer connections. Can be limited to LAN only.

Security

The silent testing framework operates by assigning tests to runners. While the runner is a static page that connects to Microsoft eCDN's backend, an executed test is dynamic and can run any target page. For that reason, runners run inside a webpage that is sandboxed by the browser and relies on security mechanisms that are built into modern browsers. Regardless of the integration (excluding custom integrations), the target page is always executed in a safe, clean context using an iframe.

The network permissions are also constrained by the browser and limited to common web APIs, including HTTP, WebSocket, WebRTC and so on.

While waiting for tests to be executed, runners maintain a persistent WebSocket connection over a secure TLS connection (WSS).

Appendix

Appendix A: How to schedule a silent test

1. Go to your Silent Testing dashboard

2. Select the + symbol

   

3. Fill in the required fields

   

   • Name - Arbitrary name of your choice.

   • Time & Date - Specific time at which the test begins.

   • Duration - The duration of the test. We recommend at least 20 minutes to allow for a proper simulation.

   • Target URL - A publicly available URL of the event page that plays video during the simulated event. You can use our built-in page or create your own.

     • Built-in Stream - Microsoft eCDN includes an already integrated built-in page with a live stream that includes several renditions and customizable streaming protocols.

     • Custom Stream - You might want to only provide a live stream of your own and use Microsoft eCDN's autointegrated page. The stream must be publicly available and include CORS headers so that the runners are able to load it. The stream is auto-played when the test begins.

     • Custom Page - A custom page of your own. The page needs to include a player and a live stream and be integrated with Microsoft eCDN. The player MUST start playing the video automatically since during the test there's no user interaction. Some browsers limit the ability to autoplay video. For that reason, it's recommended to mute the audio, which eases the limitation. The built-in pages are muted by default.

   • Device Filters - Limit a test to a specific group of devices. In some cases, you might want to run a test on a subset of the connected devices. For example, to only run a test on offices in the US or only Direct runner devices.

     • Countries Filter - Only include devices from certain countries/regions (GeoIP).

     • Integration Filter - Only include devices that are connected through a certain integration.

     • Device Id Filter - Run a test only on specific device IDs. This filter is used mostly for local debugging purposes.

4. Select Schedule and the test is created.

5. When the start time of the silent test is reached, the test will be executed on the assigned connected devices.

Appendix B: Integrating runners using headless browser

Microsoft eCDN provides a no-install silent tester script.

This script launches a chromium browser in the background of machines on a specific page for a specified duration and then closes the background browser process.

In addition, Microsoft eCDN provides a script to run the silent tester on remote machines listed in Active Directory.

Note

Restarting a machine won't automatically restore the runner and the browser will need to be launched again using the script.

Run instructions for Windows environment

1. Download silent-tester-runner-windows.ps1 - a PowerShell script that launches a chromium based browser (Microsoft Edge or Google Chrome if Edge isn't found) in the background for 24 hours.

2. Edit silent-tester-runner-windows.ps1:

   • $TenantID- Replace TENANT_ID with your Microsoft tenant ID.

   • $TestID- Replace TEST_ID with a unique ID string. This string is used in the creation of log files, allowing silent test administrators to uniquely identify test results.

   Important

   Each test must have a unique $TestID. If the script detects that it had previously ran with the same $TestID as the current instance, it will exit without executing the silent test.

   • (Optional) $scenarioDuration - Set browser uptime duration to the desired value in seconds. You are able to run silent tests on the target machines for this duration. As the browser is idle, there's no problem in increasing this value to several days to allow more flexibility in running tests. This process doesn't survive a system reboot. Defaults to 86,400 seconds (24 hours).

   • (Optional) $customChromePath - In case Microsoft Edge or Google Chrome isn't installed in the default path (C:\Program Files or C:\Program Files (x86)) set this variable to your browser's executable's path. for example: C:\Custom Path\Edge\msedge.exe

3. Execute the script on target machines using your method of choice such as one of the following options.

   • Using GPO

   • Using SCCM

   • Or Manually from a Domain Controller. For your convenience, we offer an invocation script example.

     1. Download remote-invocation.ps1 - a PowerShell script that executes silent-tester-runner-windows.ps1 on all computers in Active Directory

     2. (optional) Edit the script to limit your Active Directory query to a certain group of computers based on your needs. Consult the Get-ADComputer cmdlet documentation for advanced filtering.

     Note

     Make sure that silent-tester-runner-windows.ps1 is in the same directory you're running the invocation script from.

     Caution

     For best results, execute the runner script under a user context. Executing the runner script under the SYSTEM account is discouraged.

4. Go to the silent testing dashboard and make sure the target machines now show up as online runners.

Run instructions for Mac environment

1. Download silent-tester-runner-mac.sh - a Bash script that launches Google Chrome in the background for 24 hours.

2. Edit silent-tester-runner-mac.sh:

   1. ecdnCustomerId - Replace CUSTOMER_ID with your Microsoft tenant ID.

   2. (Optional) scenarioDuration - Set browser uptime duration to the desired value in seconds. You are able to run silent tests on the target machines for this duration. As the browser is idle, there's no problem in increasing this value to several days to allow more flexibility in running tests. Defaults to 86,400 seconds (24 hours).

3. Depending on the tool used for managing the devices at the site, Jamf Pro, for example, there are different ways to execute the script on different machines.